CitySample UE5 with AirSim(Multirotor)

[Superkart]

Hi,
Using Kamil (@czero69) plugin for UE5 support of AirSim, I was able to use the multirotor mode with Hitl. Without making any changes to the plugin, I have used the CitySample environment with Smallcity map. Gamemode as Airsimgamemode. Just a few issues I have run into with the different camera views that AirSim shows and the fpv mode is not working as expected. Will be working the fix to the cameras.
If anyone has fixed this issue please help me out.
Thanks.

[Kamil (czero69)]

Hey,
Thanks, I also saw some multirotor functionality is wokring. I am using Airsim 'computer vision mode' exclusively, and the cameras are matching 1:1 (no problem with camera mismatch). I am also using python api to control everything - some time later I will update my instructions and repos - it is still work in progress.

[Kamil (czero69)]

Hey,

I had similar issue at the beginning, and I solved it by checking/commenting UE SetWorldOrigin() or sth like that methods, at least in 'computer vision mode'.

I solved that problem, but for 'computer vision mode' only. Never checked this behaviour in 'multirotor'. There is a chance I commented out some 'multurotor' functionality to make it running - I don't remember, for that you would need to check my earliest commits.

With 'computer vision mode' + Python API, I can now control camera in an arbitrary way. I do not need anything else, this process is enough for me to collect data I need, so I do not plan to visit 'multirotor' support anytime soon.

Right now, in my python scripts there is a way to run some csv files with camera positions list and save the screenshot + gbuffers data. But I still need to clean the documentation and code a bit - these things and more YT examples will come in upcoming weeks!

Here is a slight camera movement captured via python api in 60 fps @ fullhd. Also all G-buffers from Airsim (and more) are saved too
https://www.youtube.com/watch?v=AQAvTCjn2OI

What is your use case?

[Superkart]

Thank you, i tried it and it seems to fix most of the glitches. I'm looking to use it for simulation of obstacle avoidance as of now. Maybe I will need it to simulate more applications in the future. Will probably do some rework on the cameras that show different view ports. Like a thermal view.

[Kamil (czero69)]

Glad you were able to fix most of the glitches.

Here is some recent vid showing my usecase and buffers visualization:
https://www.youtube.com/watch?v=rTLAX51vzCM&t=12s

About the Thermal it is really interesting project. One approach is to make fake thermals with UE (custom materials, shaders). Another approach would be building custom NeuralNet for RGB->IR. Ill posed problem for domain transfer, but somethihng cool to consider (as many to many mapping) if you can get variety of data in Infrared domain.

[Superkart]

Wow! Really cool. Thank you man I will try to implement it with custom shaders from unreal.
Also I'm having a little trouble setting up the hitl to use the unreal camera inputs for navigation.
Would be awesome if you have any leads on that. Thank you. I really love your work!

[Kamil (czero69)]

Hey, I guess you mean a drone camera hilt? I am not sure I will be able to help here (I didnt use it like that), but I will share you my observations.

I am setting camera from the python API, and there is one thing worth to note - Airsim coordinates notion is slightly different than UE coordinates notion.

You can check my python scripts how I deal with the camera handling, quaternions, and translating UE5->Airsim(NED) coordinates (and vice versa):
PythonClient/computer_vision/UECapture/capture.py - this script is producing video frames and gbuffers by connecting to UE5 Citysample Airsim Demo and taking camera position list as an input; target_fps param says what is desired video output fps. Time dilation says how much you want to slow down the simulation (ue5 demo), default is 100x slow down - so it can reach smooth 60fps output easily. Still need to add some kalman filter to smooth camera movemnts from the list.

check these:
getPoseAndRotation(self) toNED(self, pos, quat, scale=0.01)

in get_pose_from_camera_path(..) I am reading camera position from a .txt list (cam pos recorded with quest 2, example .txt should be on the repo already) and then I am interpolating positions for desired time intervals (output video frame-rate etc.); toNED() is transforming UE coordinates to Airsim (NED).

The code needs refactoring here and there; and more documentation - but mby you will find it usefull as it is :)
I already have a lot of changes (was fighting with velocity buffer correctnes lately), but still need to check/clean them before pushing.

[Superkart]

Thank you so much! Will definitely go through the changes you have made to use the camera. Its interesting how you have implemented it!

[Superkart]

Hey Kamil,
Lately I've been trying to get px4 data directly into unreal project by connecting to it through the given c++ api. But I into this error. If you've seen this error, it would be great if you could help out.
ran

Thanks and regards.

[Kamil (czero69)]

Hey,

When I had these errors in the past, it is because Airsim libs are builded with different (visual studio/toolkit version/something else).

1. Use these versions:

• Visual Studio 2022
• Windows 10 SDK (10.0.18362.0)

2. Rebuild Airsim from my branch. Build airsim plugin from this branch the same way as in official airsim building instructions.

3. copy a plugin dir (AirSim→Unreal→Plugins) to YourProject plugins dir (YourProject →Plugins). Build your project with the same environment/toolchain as you've built my modified version of Airsim.

4. Linker error should be resolved now.

Good Luck!

[Superkart]

Oh, thank you! Will try this solution.