Writing unicode to a file without it looking terrible

[reepblue]

I'm currently working on an application with the App Kit but I took time to make my own logging system and I didn't want my log dump file to be trash.

Not sure you've figured this out on your end. I ended up storing everything within a vector wstring container, then dumping all of its contents on shutdown. I might make the logging its own smart pointer class, so it'll dump its contents once it goes out of scope. I found it on stackoverflow and it seems to work.

//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
void CBaseApp::Log(const std::string& pszMsg)
{
	std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter;
	std::wstring result = converter.from_bytes(pszMsg);

	std::wcout << result << std::endl;
	m_vLogOutStream.push_back(result);
}

//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
void CBaseApp::Log(const std::wstring& pszMsg)
{
#if defined (WINDOWS)
	_setmode(_fileno(stdout), _O_U16TEXT);
	std::wcout << pszMsg << std::endl;
#else
	std::cout << pszMsg << std::endl;
#endif

	m_vLogOutStream.push_back(pszMsg);
}


//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
void CBaseApp::Shutdown()
{
	if (m_vLogOutStream.size() > 0)
	{
		std::wstring filename = AppName() + L" - " + CurrentDate() + L".log";
		std::wofstream outstream(filename, std::ios::out | ios_base::app | ios_base::binary);

		wchar_t buffer1[128];
		outstream.rdbuf()->pubsetbuf(buffer1, 128);
		outstream.put(0xFEFF);

		for (const auto& input : m_vLogOutStream)
		{
			outstream << input << std::endl;
		}

		outstream.close();
	}
}

int main(int argc, const char* argv[])
{
	CBaseApp* app = new CBaseApp();
	app->Log("Hello World!");
	app->Log(L"AppData Path: " + app->AppDataPath());
	app->Log(L"App Path: " + app->AppDirectory());
	app->Log(L"Привет! This is awesome.");

	for (int i = 0; i < 16; i++)
	{
		app->Log(L"Тестирование Юникода!");
	}

	app->Log("Back to english.");

	app->Shutdown();


	return 0;
}

 

My log file looks like this.

Hello World!
AppData Path: C:/Users/reepb/AppData/Local/Untitled App
App Path: C:/Users/reepb/Documents/Leadwerks/Projects/AddonCreator
Привет! This is awesome.
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Тестирование Юникода!
Back to english.

 

[reepblue]

Here's my logger class, removed the buffer completely.

//========= Copyright Reep Softworks, All rights reserved. ============//
//
// Purpose: 
//
//=====================================================================//
#include "pch.h"
#include "Logger.h"
#include "App.h"
#include <fcntl.h>

//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
CLogger::CLogger(IApp* pApp)
{
	m_vLiveStream.clear();
	m_pszFilePath.clear();
	m_pApp = NULL;

	if (pApp != nullptr)
	{
		m_pApp = pApp;
		std::wstring filename = m_pApp->AppName() + L" - " + m_pApp->CurrentDate() + L".log";
		m_pszFilePath = m_pApp->AppDataPath() + L"/" + filename;

		m_OutStream.open(m_pszFilePath, std::ios::out | ios_base::app | ios_base::binary);
		const std::locale utf8_locale = std::locale(std::locale(), new std::codecvt_utf8<wchar_t>());
		m_OutStream.imbue(utf8_locale);
#if defined (WINDOWS)
		_setmode(_fileno(stdout), _O_U16TEXT); // Might be windows exclusive??
#endif

		if (!m_OutStream.is_open())
		{
			std::wcout << L"Failed to create logger outstream: \"" << m_pszFilePath << "\"" << std::endl;
			m_pszFilePath.clear();
		}
	}
}

//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
CLogger::~CLogger()
{
	m_vLiveStream.clear();
	m_pszFilePath.clear();

	// Close the stream.
	WriteOut();

	m_pApp = NULL;
}

//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
std::vector<std::wstring> CLogger::GetLiveStream()
{
	return m_vLiveStream;
}

//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
void CLogger::Log(const std::string& pszMsg, LogType iType)
{
	std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter;
	std::wstring result = converter.from_bytes(pszMsg);
	std::wcout << result << std::endl;

	m_OutStream << result << std::endl;
	m_vLiveStream.push_back(result);
}

//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
void CLogger::Log(const std::wstring& pszMsg, LogType iType)
{
#if defined (WINDOWS)
	std::wcout << pszMsg << std::endl;
#else
	std::cout << pszMsg << std::endl;
#endif

	m_OutStream << pszMsg << std::endl;
	m_vLiveStream.push_back(pszMsg);
}

//-----------------------------------------------------------------------------
// Purpose:
//-----------------------------------------------------------------------------
void CLogger::WriteOut()
{
	// Close the stream.
	if (m_OutStream.is_open())
	{
		m_OutStream.flush();
		m_OutStream.close();
	}
}

 

[Josh]

I can't remember what character it is, but I think there is a single byte you can write at the start of a text file in Windows and it will display a WString correctly after that. You can figure this out by saving text files from VSCode or maybe Notepad with different encordings, and then read the first byte and figure out the pattern.

[reepblue]

These pages were the most helpful. Maybe it's that 0xFEFF byte they talk about?

Writing Unicode to a file in C++ - Stack Overflow

c++ wofstream issue in unicode program - Stack Overflow

 

Right now, I have a solution working but I need to battle test it still.