iOS Build Environment Help Center

ANNOUNCEMENT - status update & a call for help!

append delete Pierre-Marie Baty

Hello all,

My latest announcement on the project status was on the 17th august - cf. https://www.pmbaty.com/iosbuildenv/help/thread.php?path=Problem%20solving/&file=announcement-support-and-updates-will-be-delayed

November is here, and here is a long-awaited status update :-)

I have finished delivering a most critical contract (mostly unrelated, something around the QNX8 ecosystem) and now have some time to work on this project again.

Alright. I am assessing the situation like this :

- This toolchain need a major upgrade/rewrite that requires months of fulltime work, and it needs to happen quickly before Apple moves too far away,

- There are users of the current version of the tools who patiently await my support for weeks if not months.

Unfortunately I came to the conclusion that I won't be able to do both. I have to focus on what is the most important.

If I start reading all the support threads that require my attention (some for months) and invest my time trying to reproduce the problems (which involves downloading user-supplied Xcode projects, setting them up for building, build them in verbose/debug mode and scour the logs, try something, rebuild, try the next thing, rebuild, etc.) to provide a solution to each of them using the currently available tools, although I understand that would be the most desirable option for those who rely on this toolchain to get something done quickly, I will not have enough time to advance on rewriting the toolchain. For certain.

On the other hand, if I dedicate all my time to the much needed rewrite that I have planned, I leave my users down. This is not right, so if I choose to do this, then I should put all the sources and docs on a public repository so as to let a chance for the community to investigate the most important problems and provide patches and solutions. But if I release all my source codes to the wild wild web, the question of how to compensate (I'm not even talking about "funding") the development for the next version of this toolchain will immediately rise.

I am happy to witness that people are starting to help each other on this forum. It's a huge relief for me.

So, I am very attracted by the idea of open sourcing the whole thing and dedicating my time to proceed to the the necessary rewrite.

Yet, even if solving technical challenges that involve programming is my passion, the time I'm about to invest here will require some form of compensation to sustain the momentum. Else why not just accept other interesting programming contracts when I am offered some - considering that times are getting more and more challenging. So, before I go too far there I would like to know if it's going to be worth it: I don't want to chomp on my personal and social life for something that would prove useless or unsustainable.

Here is my (100% technical) vision for the future :

- The Apple build environment should be able to build *any* Xcode project and target *any* Apple ecosystem, not just iOS, from *any* OS (not just Windows) as long as the host sytem has virtualization support. People should be able to build Xcode projects for macOS, tvOS, visionOS, watchOS, etc., like if they were using xcodebuild on a Mac, but in a virtualized POSIX platform with these open source tools.

- For convenience, the toolchain would be distributed as a WSL2 (Windows Subsystem for Linux) transparent virtual machine, using the stock WSL2 Linux kernel supplied by Microsoft to optimize compatibility with the other WSL2 distributions already installed by the user, but this special distro would run a Darwin-compatible BSD userland (e.g. Chimera Linux, which uses the FreeBSD userland, much closer to macOS than the GNU tools are), and using a case-insensitive filesystem like macOS filesystems are.

- That system would be transformed in such a way that from a Xcode project's point of view, the typical folder hierarchy of a Mac (/Applications/Xcode.app/Contents/Developer, /Users/username, ~/Library, etc) would be present, and all the build tools that can be found on a Mac where Xcode is installed would be at identical locations (clang, plutil, ld64, codesign, etc).

- That system would provide the Apple branch of the open source LLVM compiler suite, built for this platform with all options enabled, and target an Apple system by default (just like the current toolchain does, but without the hinderances imposed by a Microsoft OS).

- All the current toolchain tools should be syntactically aligned with the Apple ones, so that they can be used as drop-in replacements in build scripts.

- The build process should take the Xcode project as input and output a codesigned, notarized, entitled deployable package (.ipa, .dmg, .app) and make it available on the user's host system (i.e. Windows). All the operations should be straightforward to perform, easier than with Xcode if possible.

- A master build script reproducing all the Xcode build steps, and able to interpret and run all the Mac tools that are present on macOS, but this time in the Chimera virtual machine, would be written in a standard scripting language that everybody can understand.

- Optional: it should be possible to upload a signed development version of a compiled app to an Apple target device for a debug session that would fire up a remote debug server with the appropriate credentials, expose a LLDB interface, and use the user's favourite IDE (as long as it can talk to a remote GDB/LLDB server, e.g. VSCode does that) to drop breakpoints and step-debug in the application source code.

- Things should be made properly and cleanly as much as possible, with diligence but without haste, to extend this project's life expectancy as far as possible. Every code block should be commented, every design choice should be justified.

It is straightforward. Really. I _already know exactly_ how to solve each and every one of the technical challenges that can be encountered in this roadmap. I already did proof of concepts for most of them.

Currently, I already have a custom Chimera Linux distribution in WSL2 which I am modifying to make it look like a macOS filesystem layout, and I am porting the build tools in it. There is work to do to align the tools names and syntaxes to the macOS ones, but it's going well. As for the case-insensitive filesystem problem, since the Linux kernel supplied by Microsoft for WSL2 doesn't support neither the ZFS filesystem nor Unicode-enabled ext4 (both would have been able to provide a "case insensitive" option), I took a few days to made a hook library that interfaces between the shell and the system's libc (using the well-known LD_PRELOAD trick), that remaps all pathnames that are passed through it by looking whether a corresponding case-insensitive equivalent exists first, and use that pathname instead when calling the real libc function. Roughly one hundred functions have been hooked this way. It works well, I am able to type things such as "cat /ETC/PASSWD" and it yields the contents of /etc/passwd for example. The advantage of such a hook library is that it has the potential to make *any* filesystem case-insensitive.

End of the technical part. Now comes the question of sustainability, because it's quite some work.

I am wondering whether a donation-based, crowdfunding-style contribution system could yield enough to compensate for the time, effort and resources I'd make into this. I am a passionate at work, but I have _zero_ experience with fundraising. This has never been an area of interest for me. And because I willingl<ctrl-left, delete> stupidly never put usage trackers nor telemetry stuff in my products (as astounding as it can sound these days), I am totally unable to assess the popularity of this toolchain _objectively_, i.e. with numbers, and thus unable to infer anything from unexistent statistics. Mind that I _do not even know_ how many people downloaded the Unity asset !

So, *I would like to call all of you who read this far and ask you, if you have some backed experience in fundraising for open source projects, or if you know some entity who's willing to sponsor that one, to contact me at my email address.* (pm at this domain). On anything technical, I think I'm good enough. On this, I don't even know where to start.

I hope I made an intelligible point here. Thank you for reading. Please spread the word where you think it could help. It's up to you :-)

Reply RSS

Replies

append delete #1. Pierre-Marie Baty

Oh and I would like to thank Epic Games for having sponsored my work last year, and particularly some wise guy working there who'll recognize himself for sure. If I'm still here working on this and am now ready to take the risk to move to the next level, it's thank to you guys. :-)

append delete #2. brian

Thank you for the update, and I hope others have the ability to help out regarding fundraising and/or programming (if you need it).

I, for one, will definitely be able to help with a donation. I have no problem donating $100 to get this toolchain up and running again. I am a game programmer, not a systems or even a Mac guy, so this tool is of utmost importance to me and my projects.

Looking to hearing back with more information when you have a clearer idea of how you are going to move forward. Thanks again for a great tool!

append delete #3. Marius

You have a couple of mainstream options for this, although I'm sure that other folks with more experience in this area would be able to add content and context.

The most prominent options are probably going with either a Kickstarter or monthly Patreon subscription model (or, both...?), with tiered rewards depending on the level of contribution. These rewards may be add-on features, named credit on a loading screen, a limited number of higher-priority support agreements, feature-requests \ -suggestions (...within reason, of course. Be VERY clear in your terms and conditions about this), discounted upgrade paths, etc.

The catch here is that you'll need to be quite active (or more specifically, VISIBLE) in the main social hubs for your target audience to gather the requisite volume of support to make this a success. So, regular and compelling posts and interactions on the Unity forums, main reddit communities, perhaps a few posts per week on a twitter feed, etc., with many of those linking back to your website and kickstarter \ Patreon \ whatever, to increase your exposure surface area. Understanding that this can be a time-consuming effort, your best bet might be to select 5 to 10 moderators from your community and social \ professional circles to help you do that in a sustainable and practically viable way. You can touch base with them once a week or so and feed them with steady updates, surveys \ polls, blog posts, community feedback requests and such to keep the momentum and the conversations going. With any luck you can each limit the actual work effort to each individual to only a few hours per week, and use the community to generate the conversations and the hype FOR you, so that you can focus on the technical work from there.

Regardless of the direction you choose, you're likely going to have to consider how to scale in a way that doesn't place all the burden of time and effort on you as an individual for the non-technical aspects of the endeavour.

With the right approach, I suspect you can make this work well for you. A fair number of Apple-targeting devs rely on your tool(s), and I'd certainly be keen to support you for the next stage in your journey as well.

append delete #4. Pierre-Marie Baty

Hello @Marius and thank you for this very insightful comment. I took one day to ponder.

Honestly, I agree with all that you said. I already suspected that the amount of "social engineering" required for this to work would be too much for one single person, especially when that person is me. Indeed, if I quit Facebook, LinkedIn and similar social networks ~10 years ago it was because the time I was spending there was as much time that I wouldn't spend doing actual work, and from the efficiency point of view alone, it has been a great choice, and allowed me to get a lot of things done. But more than that, not only I'm bad at advertising things, but I positively hate it, so I won't be the man for that sort of job. Teasing and promising feels so "alien" to me. I'm not wired for that. Thus, I need help from other people here.

At this point of my thoughts I wonder if creating a sort of non-profit association, regrouping the prominent users of these tools, those that have the most interest in keeping it going, and are technically literate enough to help steer it the right way and interface with users along with me (and accessorily pally all that I'm so bad at) and why not help with the implementation -- yeah, I think that could be the right way to go. Big projects like e.g. FreeBSD or Mozilla work this way, and I have the feeling it works quite well for them (of course, that's orders of magnitude bigger).

In such a situation, the involved people would share the donations respectively to the involvement of each, all infrastructure/running costs deduced, and all due invoices paid. There would be a board of people, a president, an accountant, there would probably be the need to produce paperwork on a regular basis - something I'm no better at btw, but I could live with that.

The only problem (!) is that, by quitting all social networks 10 years ago, there is simply no community/social/professional circle left from which I could build such a core team. The only community and social circles to which I actually belong are real-life ones and the things I've been doing here are completely alien to them.

Heh. Every choice is a renouncement...

Or, option 2, some *existing* organization decides to adopt this project. Which one could be interested ?

Yet, what you said in your last phrase is right. Even if I have no exact idea how many people are using these tools, I've had exchanges with people from Activision, Epic, Unity, and even Apple (!) about it. I was most surprised about the latter, and to my surprise about how many at Apple were using this toolchain I was replied "Haha! More than you know!" -- whatever this (very subjective) feedback says, it hints that there *is* enough interest in keeping it going.

@all, what do you think ?

Reply

(Leave this as-is, it’s a trap!)

There is no need to “register”, just enter the same name + password of your choice every time.

Pro tip: Use markup to add links, quotes and more.

Moderators: Pierre-Marie Baty