Some clarifications needed on TransmitProfile and EventPriority

[nishchith-cp]

Need these details for both iOS and Android:

1. Can we set the transmit profile to the LogManager multiple times? Eg: Set different transmit profile based on the network strength
2. What priority events will be skipped for NearRealTime and BestEffort transmit profiles?
 It was mentioned in the documentation that “Low priority events will not be transmitted, except for the RealTime transmit profile, unless it will not incur additional cost to the client application in terms of resource consumption, which mostly means when the device is on plug-in power and on a non-metered network”
3. ODWEventPriorityUnspecified -> This is the default event priority being used. In what cases events will be skipped with this priority?
4. What's the order of the event priorities for being skipped?
5. We would want to specify custom TransmitProfile instead of the default ones provided by the SDK. As per the below documentation, we need to set the timer value seconds for the associated event priorities. We would want to control these values through ECS settings.

• Is there any recommendation on the range of the timer values that can be set?
• Is there a way to test if there are any loss of events when custom transmit profiles are used?
  https://www.aria.ms/developers/how-to/custom-transmission-profile/

6. Is the above documentation valid? Is there any OneDS documentation on this?

typedef NS_ENUM(NSInteger, ODWEventPriority)
{
    ODWEventPriorityUnspecified = -1,
    ODWEventPriorityOff = 0,
    ODWEventPriorityLow = 1,
    ODWEventPriorityNormal = 2,
    ODWEventPriorityHigh = 3,
    ODWEventPriorityImmediate = 4
};

typedef NS_ENUM(NSInteger, ODWTransmissionProfile)
{
    ODWRealTime = 0,
    ODWNearRealTime = 1,
    ODWBestEffort = 2
}

cc: @aamittal1989 , @karthika-msft

[kindbe]

https://github.com/microsoft/cpp_client_telemetry/blob/master/docs/custom-transmit-profile.md has some more details about decoding the transmit profiles. Others on the issue, please jump in and correct any of the below if I'm wrong:

1. You can change the active profile at any point.
2. The timer definitions for NearRealTime and BestEffort can be found in lib/tmp/TransmitProfiles.cpp starting on line 373. If I'm reading them correctly, in both cases they are skipping EventLatency_Normal and EventLatency_CostDeferred events on metered networks.
3. In the case of ODWEventPriorityUnspecified, it is using the default value for the underlying native EventProperties object, which is EventLatency_Normal.
4. "Priority" is overloaded to mean both upload latency and storage retention. Per custom-transmit-profile.md, the first digit of the profile controls EventLatency_Normal and EventLatency_CostDeferred, the second digit is not used (but should be set to the same value as the first digit), and the final digit controls the EventLatency_Realtime upload cadence.
5. The obj-c wrapper doesn't provide a way to set a custom transmit profile; I'd recommend logging an issue (or submitting a PR). I don't know if there are formal recommendations, but I can share that we use a five minute upload cadence, and block upload for Normal/CostDeferred events on metered networks. We've tried to minimize cost to consumers both in terms of bandwidth costs and battery drain via network wakeups. In terms of testing data loss, I don't know if evt_stats captures how many events are skipped, but that might be one way. Another might be adding your own sequence number and looking for gaps.
6. That document is out of date, and I'd still start with https://github.com/microsoft/cpp_client_telemetry/blob/master/docs/custom-transmit-profile.md.

[suryatn]

@kindbe @maxgolov @lalitb @larvacea @sid-dahiya @mkoscumb - we have below follow up queries regarding custom transmit profile:

1. Can we switch from custom transmit profile to built-in transmit profile and vice versa at any point in the applifecycle?
2. In Android, we use only these 2 event latencies as per this logic: EventLatency.RealTime (for High, Immediate Priority), EventLatency.Normal (for Low, Normal Priority) i.e we do not use EventLatency.CostDeferred at all. Do we need to set the 2nd value of timer equal to the first one as per this wiki? Because as per timer definitions of built-in NearRealTime here in code, the timer values of first and second are different. We want to initially replicate exactly the built-in NearRealTime profile and want to experiment changing the timer value via ECS.
3. What happens to the events where timer is set to -1 (e.g EventLatency.Normal in metered condition), will they be stored in-memory/cache and be uploaded once the conditions are better?
4. Can we expect any gains in terms of perf/power/COGS by increasing the timer value in custom profile (e.g from 3 seconds for built-in NearRealTime profile in Unmetered charging to some other value say 5 or 10 seconds)?
5. Even as per debug logs, we are seeing that the timer values are matching to that mentioned here in code. Can we get a confirmation on what should be the ideal JSON for built-in NearRealTime transmit profile i.e if we should follow as per wiki or the code?

2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: name=NEAR_REAL_TIME
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [0] netCost= 3, powState=-1, timers=[ -1, -1, -1]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [1] netCost= 2, powState= 0, timers=[ -1, 24, 12]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [2] netCost= 2, powState= 1, timers=[ -1, 24, 12]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [3] netCost= 2, powState= 2, timers=[ -1, 18,  9]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [4] netCost= 1, powState= 0, timers=[ 24, 12,  6]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [5] netCost= 1, powState= 1, timers=[ 24, 12,  6]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [6] netCost= 1, powState= 2, timers=[ 12,  6,  3]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [7] netCost= 0, powState= 0, timers=[ 24, 12,  6]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [8] netCost= 0, powState= 1, timers=[ 24, 12,  6]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [9] netCost= 0, powState= 2, timers=[ 12,  6,  3]
2021-09-23 19:48:06.013 8007-8488/com.microsoft.skype.teams.dev D/MATSDK: [10] netCost=-1, powState=-1, timers=[ -1, -1, -1]

Please help in clarifying these queries as we want to start experimenting with custom transmit profiles in Teams Android.

cc @aamittal1989 @nishchith-cp

[lalitb]

@suryatn - I can reply to a few of your queries. Others on the issue can pitch in for other queries or correct me, else I will reply to the rest tomorrow after validating the code. Would also like to hear the result of Surya's experiments.

1. Yes, you should be able to switch across different transmit profiles ( custom or default ) in between the app processing.
3. Events are not lost or dropped as long as they don't cross the configured memory/DB limit. They are stored, and transmitted once the network condition, battery condition, or transmit profile is changed.
4. Yes, theoretically you should see a gain in perf/power with an increase in the time between uploads, best would be to profile the application against different values to see the result.

[lalitb]

@suryatn, regarding 2 and 5 queries as per the code below:

1. If the first two timer values are equal, it would be the normal transmission irrespective of the third timer value.
2. Else. If the first timer value is negative, it would be the real-time transmission.
3. Else transmission latency alternates between normal and real time.

cpp_client_telemetry/lib/tpm/TransmissionPolicyManager.cpp

Lines 358 to 378 in 0fa60bb

	EventLatency TransmissionPolicyManager::calculateNewPriority()
	{
	updateTimersIfNecessary();
	if (m_timers[0] == m_timers[1])
	{
	return EventLatency_Normal;
	}
	if (m_timers[0] < 0)
	{
	return EventLatency_RealTime;
	}
	if (m_runningLatency == EventLatency_RealTime)
	{
	return EventLatency_Normal;
	}
	return EventLatency_RealTime;
	}

So to achieve the near-realtime transmissions, the first two values shouldn't be equal, and the second value is ignored in code ( other than in the above comparison ). I would stick to the values in the code for near-real-time transmission.

Tagging @larvacea as the author of the wiki doc, if he has other/more recommedations.

[suryatn]

Thanks @lalitb. We will stick to the values mentioned in the code for NearRealTime transmission for now.

Can you also help explain the below:

2. Else. If the first timer value is negative, it would be the real-time transmission.

What would happen in case of Roaming rule where all the timers are [-1, -1, -1] (ref)? I am assuming even the EventLatency_RealTime events will not be sent over wire as its timer value is -1.

3. Else transmission latency alternates between normal and real time.

Does it mean that if the first and second timers are > 0 (ref), sometimes EventLatency_RealTime events get sent over wire and other time EventLatency_Normal events get sent and it alternates between the two? also when does it alternate?

---

Tagging @larvacea as the author of the wiki doc, if he has other/more recommedations.

@larvacea can you also please check once and provide your inputs? Tagging @maxgolov incase you have some suggestions as well.

[suryatn]

Checked with @larvacea. As per him the values mentioned in code and the format suggested in wiki should both work fine for NearRealTime transmissions. He can provide more details / context here.