pycti library has problem with non-ASCII characters

[Lhorus6]

Description

In a stream connector, we use the "self.helper.listen_stream()" function to listen to a stream. The problem is that the data is truncated when a non-ascii character passes through it.

The use case is as follows:

• I send File Observables in a stream.
• Some of my File have non-ascii characters in the "x_opencti_additional_names" field because the file has non-ascii characters in its name.
• The pycti function truncates the response at the non-ASCII character level.

Seen in the stream

Retrieved by my connector

Environment

OCTI 6.2.16

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Create a live stream with the filters "Entity type: File AND label: test-bug".
2. Create a File with only a MD5 hash (no author, no marking, etc to avoid noise in the stream).
3. Run in debug mode a stream connector listening your stream and with a breakpoint to the place where it processes the retrieved data.
4. Add in the "name" field of the File: 2020ë�� ì�°êµ¬ ì �문ì�� ë°� ì��ì��ì��ë¶�ì�¼ ê²½ë ¥ì�¬ì�� ì� ë°� 모ì§�ì��ê°�.hwp
5. Add the label "test-bug" on the File to send it in the stream.
6. Look at the connector side for the data retrieved. -> truncated data

Expected Output

Have the whole data, like what I have in my stream

[romain-filigran]

@richard-julien : Are you aware of that ? I'll try to reproduce it

[richard-julien]

I remember one case where we was not able to reproduce.
If we have a good repro case, we need to fix that :)

[Lhorus6]

Ping me for the repro case if needed @romain-filigran

[nino-filigran]

@romain-filigran & @Lhorus6 did you manage to reproduce? Is that an issue that we need to handle?

[Lhorus6]

Yes I did it myself (screenshots are mine)

[aHenryJard]

Reproduced this morning with @Megafredo : the issue is that the lib used to read SSE Event truncates the message, i.e. the 'msg.data' content at this line is already truncated (missing closing } for json data)

client-python/pycti/connector/opencti_connector_helper.py

Line 631 in b539fff

for msg in messages:

So I think that the fix must be ahead and done in filigran-sseclient lib.