Update 2022-11-04-luau-origins-and-evolution.md

Lots of wording tweaks and a couple of links.

docs/_posts/2022-11-04-luau-origins-and-evolution.md

@@ -10,15 +10,17 @@ It powers all user-generated content on Roblox, providing access to a very rich
 
 It also powers a lot of application code that Roblox engineers are writing: Universal App, the gateway to the worlds of Roblox that is used by tens of millions of people every day, has 95% of its functionality implemented in Luau, and Roblox Studio has a lot of builtin critical functionality such as part and terrain editors, marketplace browser, avatar and animation editors, material manager and more, implemented in Luau as a plugin, mostly using the same APIs that developers have access to. Every week, updates to this internal codebase that is now over 2 million lines large, are shipped to all Roblox users.
 
-But why did we use Lua in the first place, and why did we decide to pursue building our own language on top of it?
+In addition to Roblox use cases, Luau is also open-source and is seeing an increased adoption in other projects and applications.
+
+But why did we use Lua in the first place, and why did we decide to pursue building a new language on top of it?
 
 # Early beginnings
 
 Around 2006, when a very early version of the Roblox platform was developed, the question of user generated behaviors emerged. Before that, users were able to build non-interactive content on Roblox, and the only form of interaction was physics simulation. While this provided rich emergent behavior, it was hard to build gameplay on top of this: for example, to build a Capture The Flag game, you need to handle collision between players and flags spread throughout the map with a bit of logic that dictates how to adjust team points and when to remove or recreate the objects.
 
-After an early and brief misstep when we decided to add a few gameplay objects to the core definition of Roblox worlds (developers out there may recognize FlagStand as a class name...), the Roblox co-founder Erik Cassel recognized that an approach like this is fundamentally limiting the power of user generated content. It’s not enough to give creators the basic blocks on top of which to build their creations, it’s critical to expose the power of a full Turing-complete programming language. Without this, the expressive capability and the reach of the platform would have been restricted far too much.
+After an early and brief misstep when we decided to add a few gameplay objects to the core definition of Roblox worlds (some developers may recognize FlagStand as a class name...), the Roblox co-founder Erik Cassel realized that an approach like this is fundamentally limiting the power of user generated content. It’s not enough to give creators the basic blocks on top of which to build their creations, it’s critical to expose the power of a full Turing-complete programming language. Without this, the expressive capability and the reach of the platform would have been restricted far too much.
 
-But which programming language do you choose? This is where [Lua](https://lua.org/), which was, and still is, one of the dominant programming languages used in video games, comes in.
+But which programming language to choose? This is where [Lua](https://lua.org/), which was, and still is, one of the dominant programming languages used in video games, comes in.
 
 In addition to its simplicity, which made the language easy to learn and get productive in, Lua was the fastest scripting language compared to popular alternatives like Python or JavaScript at the time[^1], designed to be embedded which meant an easy ability to expose APIs from the host application to the scripts as well as high degree of execution control from the host, and implemented coroutines, a very powerful concurrency primitive that allowed to easily and intuitively script behaviors for independent actors in game using linear control flow.
 
@@ -32,9 +34,9 @@ Being a simple language means having a small set of features. Lua has all the fu
 
 Being a simple language means having a minimal set of rules for every feature. Lua does deviate from this in certain respects (which is to say, the language could have been even simpler!), but notably for a dynamic language the behavior of fundamental operators is generally easy to explain and unsurprising - for example, two values in Lua are equal iff they have the same type and the same value, as such `0 == “0”` is `false`; as another example, `for` loops introduce unique variable bindings on every iteration, as such capturing the iteration variable in a closure produces unique values. These decisions lead to more concise and efficient implementation and eliminate a class of bugs in programs.
 
-Being a simple language means having a small implementation. This may be immaterial to people writing code in the language, but it means an implementation that can be of higher quality; simpler implementations can also be easier to optimize for memory or performance, and are easier to build upon.
+Being a simple language means having a small implementation. This may be immaterial to people writing code in the language, but it leads to an implementation that can be of higher quality; simpler implementations can also be easier to optimize for memory or performance, and are easier to build upon.
 
-Developers on the Roblox platform have very diverse programming backgrounds. Some are writing their first line of code in Roblox Studio, while others have computer science degrees and experience working in multiple different programming languages. While it’s always possible to support two different programming languages that target different segments of the audience, that fragments the ecosystem and makes the programming story less consistent (impacting documentation, tutorials, code reuse, ability for community members to help each other write code, presents challenges with interaction between different languages in the same experience and more). A better outcome is one where a single language can serve both audiences - this requires a language that strikes a perfect balance between simplicity and generality, and while Lua isn’t perfect here, it’s great as a foundation for a language like this[^2].
+Developers on the Roblox platform have very diverse programming backgrounds. Some are writing their first line of code in Roblox Studio, while others have computer science degrees and experience working in multiple different programming languages. While it’s always possible to support two different programming languages that target different segments of the audience, that fragments the ecosystem and makes the programming story less consistent (impacting documentation, tutorials, code reuse, ability for community members to help each other write code, presents challenges with interaction between different languages in the same experience and more). A better outcome is one where a single language can serve both audiences - this requires a language that strikes a balance between simplicity and generality, and while Lua isn’t perfect here, it’s great as a foundation for a language like this[^2].
 
 In many ways, Lua is simultaneously simple and pragmatic: many parts of the language are difficult to make much better without a lot of added complexity, but at the same time it requires little in the way of extra functionality to be able to solve problems efficiently. That said, no language is perfect, and within several areas of Lua we felt that the tradeoffs weren’t quite right for our use case.
 
@@ -68,9 +70,9 @@ This means that when designing language features, we make sure that they can be
 
 This avoids some classes of design problems, for example we won’t specify a language feature that has a prohibitively high implementation cost, as it violates our simplicity criteria, or that is impractical to implement efficiently, as that would create a performance hazard. This also means that when implementing various components of the language we cross-check the concerns and applicability of these across the entire stack - for example, we’ve reworked our auto-complete system to use the same type inference engine that the type checking / analysis tools use, which had immense benefits for the experience of editing code, but also applied significant back pressure on the type inference itself, forcing us to improve it substantially and fix a lot of corner cases that would otherwise have lingered unnoticed.
 
-Whenever we develop features, optimizations, improve our analysis engine or enhance the standard libraries, we also heavily rely on code written in Luau to validate our hypotheses. For example, when working on new features we find motivation in the real problems that we see our developers face. For example, when implementing the [new ternary operator](https://luau-lang.org/syntax#if-then-else-expressions), we did it after seeing a large set of cases where existing Lua’s `a and b or c` stand-in was error-prone for boolean values, which made it easy to accidentally introduce a mistake that was hard to identify automatically. All optimizations and new analysis features are validated on our internal 2M LOC codebase before being rolled out to Roblox developers, which allows us to quickly get initial validation of ideas, or invalidate some approaches as infeasible / unhelpful.
+Whenever we develop features, optimizations, improve our analysis engine or enhance the standard libraries, we also heavily rely on code written in Luau to validate our hypotheses. When working on new features we find motivation in the real problems that we see our developers face. For example, we implemented the [new ternary operator](https://luau-lang.org/syntax#if-then-else-expressions) after seeing a large set of cases where existing Lua’s `a and b or c` pattern was error-prone for boolean values, which made it easy to accidentally introduce a mistake that was hard to identify automatically. All optimizations and new analysis features are validated on our internal 2M LOC codebase before being added to Luau, which allows us to quickly get initial validation of ideas, or invalidate some approaches as infeasible / unhelpful.
 
-In addition to that, while we don’t have direct access to community-developed source code for privacy reasons, we can run experiments and collect telemetry, which also helps us make decisions regarding backwards compatibility. Due to [Hyrum’s law](https://www.hyrumslaw.com/), technically any change in the language or libraries, no matter how small, would be backwards incompatible - instead we adopt the notion of pragmatic balance between strict backwards compatibility[^4] and pragmatic compatibility concerns. For example, later versions of Lua make some library functions like `table.insert`/`table.remove` more strict wrt how they handle out of range indices. We have evaluated this change for compatibility by collecting telemetry on the use of out of range indices in these functions on the Roblox platform and concluded that applying the stricter checking would break existing programs, and instead had to slightly adjust the rules for out of range behavior in ways that was benign for all existing code but prevented catastrophic performance degradation for large out of range indices. Because we couldn’t afford to introduce new runtime errors in this case, we also added a set of linting rules to our analysis engine to flag potential misuse of `table.insert`/`table.remove` before the code ever gets to run - this diagnostics is informational and as such doesn’t affect backwards compatibility, but does help prevent mistakes.
+In addition to that, while we don’t have direct access to community-developed source code for privacy reasons, we can run experiments and collect telemetry[^4], which also helps us make decisions regarding backwards compatibility. Due to [Hyrum’s law](https://www.hyrumslaw.com/), technically any change in the language or libraries, no matter how small, would be backwards incompatible - instead we adopt the notion of pragmatic balance between strict backwards compatibility[^5] and pragmatic compatibility concerns. For example, later versions of Lua make some library functions like `table.insert`/`table.remove` more strict with how they handle out of range indices. We have evaluated this change for compatibility by collecting telemetry on the use of out of range indices in these functions on the Roblox platform and concluded that applying the stricter checking would break existing programs, and instead had to slightly adjust the rules for out of range behavior in ways that was benign for existing code but prevented catastrophic performance degradation for large out of range indices. Because we couldn’t afford to introduce new runtime errors in this case, we also added a set of linting rules to our analysis engine to flag potential misuse of `table.insert`/`table.remove` before the code ever gets to run - this diagnostics is informational and as such doesn’t affect backwards compatibility, but does help prevent mistakes.
 
 There are also cases where this co-design approach prevents introduction of features that can lead to easy misuse, which can be difficult to see in the design of the feature itself, but becomes more apparent when you consider features in context of the entire ecosystem. This is a good thing - it means co-design acts as a forcing function on the language simplicity and makes it easier to flag potential bad interactions between different language features, or language features and tooling, or language features and existing programming patterns that are in widespread use in real-world code. By making sure that all features are validated for their impact across the stack and on code written in Luau, we ultimately get a better, simpler and more cohesive language.
 
@@ -80,15 +82,15 @@ One of the critical goals in front of Luau is efficiency, both from the performa
 
 Crucially, our performance needs are somewhat unique and require somewhat unique solutions. 
 
-We need Luau to run on many platforms where native code generation is either prohibited by the platform vendor or impractical due to tight memory constraints. As such, in terms of execution performance it’s critical that we have a very fast interpreter[^5]. However, we have freedom in terms of high level design of the entire stack - for example, clients never see the source code of the scripts as all compilation to bytecode happens on the server; this gives us an opportunity to perform more involved and expensive optimizations during that process as well as have the smallest possible startup time on the client without complex pre-parse steps. For example, our bytecode compiler performs a series of high level optimizations including function inlining and loop unrolling that in other dynamic languages is often left to the just-in-time compiler.
+We need Luau to run on many platforms where native code generation is either prohibited by the platform vendor or impractical due to tight memory constraints. As such, in terms of execution performance it’s critical that we have a very fast interpreter[^6]. However, we have freedom in terms of high level design of the entire stack - for example, clients never see the source code of the scripts as all compilation to bytecode happens on the server; this gives us an opportunity to perform more involved and expensive optimizations during that process as well as have the smallest possible startup time on the client without complex pre-parse steps. Notably, our bytecode compiler performs a series of high level optimizations including function inlining and loop unrolling that in other dynamic languages is often left to the just-in-time compiler.
 
 Another area where performance is critical is garbage collection. Garbage collection is crucial for the language’s simplicity as it makes memory management easier to reason about, but it does require a substantial amount of implementation effort to keep it efficient. For Roblox and for any other game engine or interactive simulation, latency is critical and so our collector is heavily optimized for that - to the extent possible collection is incremental and stop-the-world pauses are very brief. Another part of the performance story here however is the language and data structure design - by making sure that core data types are efficient in how they are laid out in memory we reduce the amount of work garbage collector takes to trace the heap, and, as another example of co-design, we try to make sure that language features are conscious of the impact they have on memory and garbage collection efficiency.
 
-However, from a whole-platform standpoint there’s a lot of performance aspects that go beyond single-threaded execution. This is an active area of research and development for the team, as to really leverage the hardware the code is running on we need to think about SIMD, hardware thread utilization as well as running code in a cluster of nodes. These considerations inform current and future development of the runtime and the language (for example, our runtime now supports efficient operations on short SIMD vectors even in interpreted mode, and the VM is fairly lightweight to instantiate which makes running many VMs per core practical, with message passing or access to shared Roblox data model used to make gameplay features feasible to implement), but we’re definitely in the early days here - our first implementation of parallel code execution just shipped earlier this year. This is likely the area where a lot of future innovations will happen as well.
+However, from a whole-platform standpoint there’s a lot of performance aspects that go beyond single-threaded execution. This is an active area of research and development for the team, as to really leverage the hardware the code is running on we need to think about SIMD, hardware thread utilization as well as running code in a cluster of nodes. These considerations inform current and future development of the runtime and the language (for example, our runtime now supports efficient operations on short SIMD vectors even in interpreted mode, and the VM is fairly lightweight to instantiate which makes running many VMs per core practical, with message passing or access to shared Roblox data model used to make gameplay features feasible to implement), but we’re definitely in the early days here - our first implementation of parallel script execution in Roblox just [shipped earlier this year](https://devforum.roblox.com/t/full-release-of-parallel-luau-v1/1836187). This is likely the area where a lot of future innovations will happen as well.
 
 # Future
 
-We’re very happy with the success of Luau - in several years we’ve established consistent processes for evolving the language and so far we found a good balance between simplicity, ease of use, performance and robustness of the language, its implementation and the tooling surrounding it. The language keeps continuously evolving but at a pace that is easy to stay on top of - in 2022 we shipped a few syntactic extensions for type annotations but no changes to the syntax of the language outside of types, and only one major [semantic change to the for loop iteration](https://luau-lang.org/syntax#generalized-iteration) that actually made the language easier to use by avoiding the need to specify the table traversal style via pairs/ipairs. We try to make sure that the features are general and provide enough extensibility so that libraries can be built on top of the language to make it easier to write code, while also making it practical to use the language without complex supporting frameworks.
+We’re very happy with the success of Luau - in several years we’ve established consistent processes for evolving the language and so far we found a good balance between simplicity, ease of use, performance and robustness of the language, its implementation and the tooling surrounding it. The language keeps continuously evolving but at a pace that is easy to stay on top of - in 2022 we shipped a few syntactic extensions for type annotations but no changes to the syntax of the language outside of types, and only one major [semantic change to the for loop iteration](https://luau-lang.org/syntax#generalized-iteration) that actually made the language easier to use by avoiding the need to specify the table traversal style via `pairs`/`ipairs`. We try to make sure that the features are general and provide enough extensibility so that libraries can be built on top of the language to make it easier to write code, while also making it practical to use the language without complex supporting frameworks.
 
 There’s still a lot of ground to cover, and we’ll be working on Luau for years to come. We’re in the process of building the next version of our type inference / checking engine to make sure that all users of the language regardless of their expertise benefit from it, we’ve started investing in native code generation as we’re reaching the limits of interpreted performance (although some exciting opportunities for compiler optimization are still on the horizon), and there’s still a lot of hard design and implementation work ahead of us for some important language features and standard libraries. And as mentioned, our execution model will likely see a lot of innovation as we push the boundaries of hardware utilization across cores and nodes.
 
@@ -97,5 +99,6 @@ Overall, Luau is like an iceberg - the surface is simple to learn and use, but i
 [^1]: High-performance JavaScript engines didn’t exist at the time! LuaJIT was around the corner and redefined the performance expectations of dynamic languages.
 [^2]: In fact, scaling to large teams of expert programmers is one of the core motivations behind our creating Luau, while a requirement to still be suitable for beginner programmers guides our evolution direction.
 [^3]: This would have been difficult to drive in any existing large established language like JavaScript or Python.
-[^4]: Which we do follow in some areas, such as syntactic compatibility - all existing programs that parse must continue to parse the same way as the language evolves.
-[^5]: Some design decisions and implementation techniques are documented on our [performance page](https://luau-lang.org/performance).
+[^4]: This is limited to Roblox platform and doesn't exist in open-source releases.
+[^5]: Which we do follow in some areas, such as syntactic compatibility - all existing programs that parse must continue to parse the same way as the language evolves.
+[^6]: Some design decisions and implementation techniques are documented on our [performance page](https://luau-lang.org/performance).