Add multi-processor support for guests

[jinjing750629]

Dear Sir

I try to send a SMP parameter to qemu bundled togehter with s2e, but got an error like this:

Number of SMP cpus requested exceeds the recommended cpus support by KVM

Does this mean s2e only support one virtual cpu?

Thanks

[vitalych]

You can use only one core in S2E.

[jinjing750629]

Currently I am fuzzing a multi thread application ( 100+ thread ) inside s2e, the speed is very slow which causes some of the thread quit abnormally. I am wondering whether there is a solution to make s2e support multiple virtual cpu?

[vitalych]

The S2E engine is single-cpu by design. I estimate it would take 6 months full time work to make it multi-cpu. We would need the following:

1. Update the KVM interface to expose multiple CPUs
2. Support multiple CPUs in S2EExecutionState
3. Make everything thread-safe (each CPU runs on one thread) and properly synchronized (S2EExecutor, LLVM backend, KLEE, etc.) while maintaining acceptable performance (simple case is to have a global lock, but that would kill all parallelism).

[jinjing750629]

These days I am trying to understand more about s2e source code to figure out what exactly we have to do to make the code thread safe. I found the overall architecture is very complicated. I am stuck inside the memory management code. There are a list of helper functions which are supposed to be called from the translated code. But I can't find the defintion anywhere. I guess they should be from softmmu_template.h, but when I look at the function body closely I realize there must be another defintion because the translated code have no idea about TB, so they should never call tb_fill. Would you please shed some lights on this?

Thanks

uint8_t helper_ldb_mmu(CPUArchState *env, target_ulong addr, int mmu_idx, void *retaddr);
void helper_stb_mmu(CPUArchState *env, target_ulong addr, uint8_t val, int mmu_idx, void *retaddr);
uint16_t helper_ldw_mmu(CPUArchState *env, target_ulong addr, int mmu_idx, void *retaddr);
void helper_stw_mmu(CPUArchState *env, target_ulong addr, uint16_t val, int mmu_idx, void *retaddr);
uint32_t helper_ldl_mmu(CPUArchState *env, target_ulong addr, int mmu_idx, void *retaddr);
void helper_stl_mmu(CPUArchState *env, target_ulong addr, uint32_t val, int mmu_idx, void *retaddr);
uint64_t helper_ldq_mmu(CPUArchState *env, target_ulong addr, int mmu_idx, void *retaddr);
void helper_stq_mmu(CPUArchState *env, target_ulong addr, uint64_t val, int mmu_idx, void *retaddr);

[vitalych]

Before you embark on a complex task, there may be a simpler thing that would solve your timeout issues, which is to slow down the guest clock. This way, the guest will think that time goes much slower and should time out less often. Use the --clock-slow-down-concrete option for this: https://github.com/S2E/s2e/blob/master/libs2ecore/src/S2EExecutor.cpp#L165.
You can set it in s2e-config.lua in the kleeArgs = {...} section.

Regarding memory helpers, these are tricky indeed, they are inherited from QEMU. softmmu_{template|header}.h contain their definition, which is obfuscated under a layer of glue() macros. Here are a few tips to understand:

• Generate a preprocessed file for op_helper.c, so that you can see how these macros are put together.
• Look around here: https://github.com/S2E/s2e/blob/master/libcpu/src/target-i386/op_helper.c#L4932
  This is where the template functions are instantiated.
• The code generator creates calls to them here: https://github.com/S2E/s2e/blob/master/libtcg/src/tcg-ldst.inc.c#L43

[jinjing750629]

Vitalych, thanks for your prompt response. I thought I understand memory helpers better following your instructions. I saw io_read/write/check will call "switch_to_symbolic", but ld/st will not. Looks like these memory helpers just read/write memory concretely. If this is the case, when will s2e transtion to symbolic execution?

[vitalych]

A memory page that contains symbolic data gets a special flag in the TLB. When ld/st see this flag, they eventually execute io_read/write/check, which will then switch to symbolic mode.

[jinjing750629]

Is the special flag you mentioned TLB_SYM? from the code ,looks like the page contains symbolic value has notdirty ops?

[vitalych]

Yes, that's right. In normal QEMU, there is notdirty_mem_ops that handles the first write access to a mapped page to set various dirty bits. S2E builds on top of that to also handle reads.

[jinjing750629]

Vitalych, S2E switches to/from a bunch of execution states. How is S2E managing the memory for these states? What will happen if two states modify contents located in the same guest vitual address? Thanks

[vitalych]

There are two steps:

1. Convert an address from guest virtual memory to host virtual memory. This is the same as on QEMU.
2. Convert the host virtual memory address to another host virtual memory address that points to the right ObjectState. This is the same as on normal KLEE.

S2E does step 2 by adding an se_addend field to the TLBEntry structure. This field is added to the guest virtual memory to directly obtain the correct address in an ObjectState without any additional overhead. Check softmmu_*.h files to see how this is done.

Two states that write to the same virtual address will trigger copy-on-write. Initially, both states share the same ObjectState. This object will be cloned on the first write and the TLBEntry of the state that writes will be adjusted to point to the cloned, private copy of ObjectState.