Update 2022-11-04-luau-origins-and-evolution.md

Add many more links

docs/_posts/2022-11-04-luau-origins-and-evolution.md

@@ -52,11 +52,11 @@ What we’ve realized is that Lua is a great foundation for a perfect language t
 
 We would maintain backwards compatibility with Lua 5.1 but evolve the language from there; sometimes this means taking later features from Lua that don’t conflict with the existing language or our design values, sometimes this means innovating beyond what Lua has done. Crucially, we must maintain the balance between simplicity and power - we still value simplicity, we still need to avoid a feature explosion to ensure that the features compose and are of high quality, and we still need the language to be a good fit for beginners.
 
-One of the largest limitations that we’ve seen is the lack of type checking making it easy to make mistakes in large code bases, as such support for type checking was a requirement for Luau. However, it’s important that the type checker is mostly transparent to the developers who don’t want to invest the time to learn it - anything else would change the learning curve too much for the language to be suitable for beginners. As such, we’ve investing in gradual typing, and our type checker is learning to strike a balance between inferring useful types for completely untyped programs (which, among other things, greatly enhances editing experience through type-aware autocomplete), and the lack of false positive diagnostics that can be confusing and distracting.
+One of the largest limitations that we’ve seen is the lack of type checking making it easy to make mistakes in large code bases, as such [support for type checking](https://luau-lang.org/typecheck) was a requirement for Luau. However, it’s important that the type checker is mostly transparent to the developers who don’t want to invest the time to learn it - anything else would change the learning curve too much for the language to be suitable for beginners. As such, we’ve investing in gradual typing, and our type checker is learning to strike a balance between inferring useful types for completely untyped programs (which, among other things, greatly enhances editing experience through type-aware autocomplete), and the lack of false positive diagnostics that can be confusing and distracting.
 
-While we did need to introduce extra syntax to the language - most notably, to support optional type annotations - it was important for us to maintain the cohesion of the overall syntax. We aren’t seeking to make a new language with a syntax alien to Lua programmers - Luau programs are still recognizably Lua, and to the extent possible we try to avoid new syntactic features. In a sense, we still want the syntax, semantics, and the runtime to be simple and minimal - but at the same time we have important problems to solve with respect to ergonomics, robustness and performance of the language, and solving some of them requires having slightly more complex syntax, semantics, or implementation.
+While we did need to introduce [extra syntax](https://luau-lang.org/syntax) to the language - most notably, to support optional type annotations - it was important for us to maintain the cohesion of the overall syntax. We aren’t seeking to make a new language with a syntax alien to Lua programmers - Luau programs are still recognizably Lua, and to the extent possible we try to avoid new syntactic features. In a sense, we still want the syntax, semantics, and the runtime to be simple and minimal - but at the same time we have important problems to solve with respect to ergonomics, robustness and performance of the language, and solving some of them requires having slightly more complex syntax, semantics, or implementation.
 
-So in finding ways to evolve Luau, we strive to design features that feel like they would be at home in Lua. At the same time, we’ve adopted a more open evolution process - the language development is driven through RFCs that are designs open to the public that anyone can contribute to - this is in contrast with Lua, which has a very closed development process, and is one of the reasons why it would have been difficult for us to keep using Lua as we wouldn’t get a say in its development. At the same time, to ensure the design criterias are met, it’s important that the Luau development team at Roblox maintains a final say over design and implementation of the language[^3], while taking the community's proposals and input into consideration.
+So in finding ways to evolve Luau, we strive to design features that feel like they would be at home in Lua. At the same time, we’ve adopted a more open evolution process - the language development is driven [through RFCs](https://github.com/Roblox/luau/blob/master/rfcs/README.md) that are designs open to the public that anyone can contribute to - this is in contrast with Lua, which has a very closed development process, and is one of the reasons why it would have been difficult for us to keep using Lua as we wouldn’t get a say in its development. At the same time, to ensure the design criterias are met, it’s important that the Luau development team at Roblox maintains a final say over design and implementation of the language[^3], while taking the community's proposals and input into consideration.
 
 # Importance of co-design
 
@@ -68,11 +68,11 @@ This means that when designing language features, we make sure that they can be
 
 This avoids some classes of design problems, for example we won’t specify a language feature that has a prohibitively high implementation cost, as it violates our simplicity criteria, or that is impractical to implement efficiently, as that would create a performance hazard. This also means that when implementing various components of the language we cross-check the concerns and applicability of these across the entire stack - for example, we’ve reworked our auto-complete system to use the same type inference engine that the type checking / analysis tools use, which had immense benefits for the experience of editing code, but also applied significant back pressure on the type inference itself, forcing us to improve it substantially and fix a lot of corner cases that would otherwise have lingered unnoticed.
 
-Whenever we develop features, optimizations, improve our analysis engine or enhance the standard libraries, we also heavily rely on code written in Luau to validate our hypotheses. For example, when working on new features we find motivation in the real problems that we see our developers face. For example, when implementing the new ternary operator, we did it after seeing a large set of cases where existing Lua’s `a and b or c` stand-in was error-prone for boolean values, which made it easy to accidentally introduce a mistake that was hard to identify automatically. All optimizations and new analysis features are validated on our internal 2M LOC codebase before being rolled out to Roblox developers, which allows us to quickly get initial validation of ideas, or invalidate some approaches as infeasible / unhelpful.
+Whenever we develop features, optimizations, improve our analysis engine or enhance the standard libraries, we also heavily rely on code written in Luau to validate our hypotheses. For example, when working on new features we find motivation in the real problems that we see our developers face. For example, when implementing the [new ternary operator](https://luau-lang.org/syntax#if-then-else-expressions), we did it after seeing a large set of cases where existing Lua’s `a and b or c` stand-in was error-prone for boolean values, which made it easy to accidentally introduce a mistake that was hard to identify automatically. All optimizations and new analysis features are validated on our internal 2M LOC codebase before being rolled out to Roblox developers, which allows us to quickly get initial validation of ideas, or invalidate some approaches as infeasible / unhelpful.
 
-In addition to that, while we don’t have direct access to community-developed source code for privacy reasons, we can run experiments and collect telemetry, which also helps us make decisions regarding backwards compatibility. Due to Hyrum’s law, technically any change in the language or libraries, no matter how small, would be backwards incompatible - instead we adopt the notion of pragmatic balance between strict backwards compatibility[^4] and pragmatic compatibility concerns. For example, later versions of Lua make some library functions like `table.insert`/`table.remove` more strict wrt how they handle out of range indices. We have evaluated this change for compatibility by collecting telemetry on the use of out of range indices in these functions on the Roblox platform and concluded that applying the stricter checking would break existing programs, and instead had to slightly adjust the rules for out of range behavior in ways that was benign for all existing code but prevented catastrophic performance degradation for large out of range indices. Because we couldn’t afford to introduce new runtime errors in this case, we also added a set of linting rules to our analysis engine to flag potential misuse of `table.insert`/`table.remove` before the code ever gets to run - this diagnostics is informational and as such doesn’t affect backwards compatibility, but does help prevent mistakes.
+In addition to that, while we don’t have direct access to community-developed source code for privacy reasons, we can run experiments and collect telemetry, which also helps us make decisions regarding backwards compatibility. Due to [Hyrum’s law](https://www.hyrumslaw.com/), technically any change in the language or libraries, no matter how small, would be backwards incompatible - instead we adopt the notion of pragmatic balance between strict backwards compatibility[^4] and pragmatic compatibility concerns. For example, later versions of Lua make some library functions like `table.insert`/`table.remove` more strict wrt how they handle out of range indices. We have evaluated this change for compatibility by collecting telemetry on the use of out of range indices in these functions on the Roblox platform and concluded that applying the stricter checking would break existing programs, and instead had to slightly adjust the rules for out of range behavior in ways that was benign for all existing code but prevented catastrophic performance degradation for large out of range indices. Because we couldn’t afford to introduce new runtime errors in this case, we also added a set of linting rules to our analysis engine to flag potential misuse of `table.insert`/`table.remove` before the code ever gets to run - this diagnostics is informational and as such doesn’t affect backwards compatibility, but does help prevent mistakes.
 
-There are also cases where this co-design approach prevents introduction of features that can lead to easy misuse, which can be difficult to see in the design of the feature itself, but becomes more apparent when you consider features in context of the entire ecosystem. This is a good thing - it means co-design acts as a forcing function on the language simplicity and makes it easier to flag potential bad interactions between different language features, or language features and tooling, or language features and existing programming patterns that are in widespread use in real-world code. By making sure that all features are validated for their impact across the stack and on code written in Luau, we ultimately get a better, simpler and more cohesive language. 
+There are also cases where this co-design approach prevents introduction of features that can lead to easy misuse, which can be difficult to see in the design of the feature itself, but becomes more apparent when you consider features in context of the entire ecosystem. This is a good thing - it means co-design acts as a forcing function on the language simplicity and makes it easier to flag potential bad interactions between different language features, or language features and tooling, or language features and existing programming patterns that are in widespread use in real-world code. By making sure that all features are validated for their impact across the stack and on code written in Luau, we ultimately get a better, simpler and more cohesive language.
 
 # Efficient execution
 
@@ -80,21 +80,22 @@ One of the critical goals in front of Luau is efficiency, both from the performa
 
 Crucially, our performance needs are somewhat unique and require somewhat unique solutions. 
 
-We need Luau to run on many platforms where native code generation is either prohibited by the platform vendor or impractical due to tight memory constraints. As such, in terms of execution performance it’s critical that we have a very fast interpreter. However, we have freedom in terms of high level design of the entire stack - for example, clients never see the source code of the scripts as all compilation to bytecode happens on the server; this gives us an opportunity to perform more involved and expensive optimizations during that process as well as have the smallest possible startup time on the client without complex pre-parse steps. For example, our bytecode compiler performs a series of high level optimizations including function inlining and loop unrolling that in other dynamic languages is often left to the just-in-time compiler.
+We need Luau to run on many platforms where native code generation is either prohibited by the platform vendor or impractical due to tight memory constraints. As such, in terms of execution performance it’s critical that we have a very fast interpreter[^5]. However, we have freedom in terms of high level design of the entire stack - for example, clients never see the source code of the scripts as all compilation to bytecode happens on the server; this gives us an opportunity to perform more involved and expensive optimizations during that process as well as have the smallest possible startup time on the client without complex pre-parse steps. For example, our bytecode compiler performs a series of high level optimizations including function inlining and loop unrolling that in other dynamic languages is often left to the just-in-time compiler.
 
 Another area where performance is critical is garbage collection. Garbage collection is crucial for the language’s simplicity as it makes memory management easier to reason about, but it does require a substantial amount of implementation effort to keep it efficient. For Roblox and for any other game engine or interactive simulation, latency is critical and so our collector is heavily optimized for that - to the extent possible collection is incremental and stop-the-world pauses are very brief. Another part of the performance story here however is the language and data structure design - by making sure that core data types are efficient in how they are laid out in memory we reduce the amount of work garbage collector takes to trace the heap, and, as another example of co-design, we try to make sure that language features are conscious of the impact they have on memory and garbage collection efficiency.
 
-However, from a whole-platform standpoint there’s a lot of performance aspects that go beyond single-thread execution. This is an active area of research and development for the team, as to really leverage the hardware the code is running on we need to think about SIMD, hardware thread utilization as well as running code in a cluster of nodes. These considerations inform current and future development of the runtime and the language (for example, our runtime now supports efficient operations on short SIMD vectors even in interpreted mode, and the VM is fairly lightweight to instantiate which makes running many VMs per core practical, with message passing or access to shared Roblox data model used to make gameplay features feasible to implement), but we’re definitely in the early days here - our first implementation of parallel code execution just shipped earlier this year. This is likely the area where a lot of future innovations will happen as well.
+However, from a whole-platform standpoint there’s a lot of performance aspects that go beyond single-threaded execution. This is an active area of research and development for the team, as to really leverage the hardware the code is running on we need to think about SIMD, hardware thread utilization as well as running code in a cluster of nodes. These considerations inform current and future development of the runtime and the language (for example, our runtime now supports efficient operations on short SIMD vectors even in interpreted mode, and the VM is fairly lightweight to instantiate which makes running many VMs per core practical, with message passing or access to shared Roblox data model used to make gameplay features feasible to implement), but we’re definitely in the early days here - our first implementation of parallel code execution just shipped earlier this year. This is likely the area where a lot of future innovations will happen as well.
 
 # Future
 
-We’re very happy with the success of Luau - in several years we’ve established consistent processes for evolving the language and so far we found a good balance between simplicity, ease of use, performance and robustness of the language, its implementation and the tooling surrounding it. The language keeps continuously evolving but at a pace that is easy to stay on top of - in 2022 we shipped a few syntactic extensions for type annotations but no changes to the syntax of the language outside of types, and only one major semantic change to the for loop iteration that actually made the language easier to use by avoiding the need to specify the table traversal style via pairs/ipairs. We try to make sure that the features are general and provide enough extensibility so that libraries can be built on top of the language to make it easier to write code, while also making it practical to use the language without complex supporting frameworks.
+We’re very happy with the success of Luau - in several years we’ve established consistent processes for evolving the language and so far we found a good balance between simplicity, ease of use, performance and robustness of the language, its implementation and the tooling surrounding it. The language keeps continuously evolving but at a pace that is easy to stay on top of - in 2022 we shipped a few syntactic extensions for type annotations but no changes to the syntax of the language outside of types, and only one major [semantic change to the for loop iteration](https://luau-lang.org/syntax#generalized-iteration) that actually made the language easier to use by avoiding the need to specify the table traversal style via pairs/ipairs. We try to make sure that the features are general and provide enough extensibility so that libraries can be built on top of the language to make it easier to write code, while also making it practical to use the language without complex supporting frameworks.
 
 There’s still a lot of ground to cover, and we’ll be working on Luau for years to come. We’re in the process of building the next version of our type inference / checking engine to make sure that all users of the language regardless of their expertise benefit from it, we’ve started investing in native code generation as we’re reaching the limits of interpreted performance (although some exciting opportunities for compiler optimization are still on the horizon), and there’s still a lot of hard design and implementation work ahead of us for some important language features and standard libraries. And as mentioned, our execution model will likely see a lot of innovation as we push the boundaries of hardware utilization across cores and nodes.
 
-Overall, Luau is like an iceberg - the surface is simple to learn and use, but it hides the tremendous amount of careful design, engineering and attention to detail, and we plan to continue to invest in it while trying to keep the outer surface comparatively small.
+Overall, Luau is like an iceberg - the surface is simple to learn and use, but it hides the tremendous amount of careful design, engineering and attention to detail, and we plan to continue to invest in it while trying to keep the outer surface comparatively small. We're excited to see how far we can take it!
 
 [^1]: High-performance JavaScript engines didn’t exist at the time! LuaJIT was around the corner and redefined the performance expectations of dynamic languages.
 [^2]: In fact, scaling to large teams of expert programmers is one of the core motivations behind our creating Luau, while a requirement to still be suitable for beginner programmers guides our evolution direction.
 [^3]: This would have been difficult to drive in any existing large established language like JavaScript or Python.
 [^4]: Which we do follow in some areas, such as syntactic compatibility - all existing programs that parse must continue to parse the same way as the language evolves.
+[^5]: Some design decisions and implementation techniques are documented on our [performance page](https://luau-lang.org/performance).