Draft RFC on function overload resolution

rfcs/function-overload-resolution.md

@@ -0,0 +1,112 @@
+# Function overload resolution
+
+## Summary
+
+The current algorithm for function overload resolution is designed for
+greedy type inference. This RFC updates it for local type inference.
+
+## Motivation
+
+The algorithm for function overload resolution determines which type
+to give `f(x)` for overloaded functions `f`. Currently it is as
+follows (simplified, for example specializing to one argument and
+result):
+
+* Let `F` by the type of `f`, and `X` be the type of `x`.
+* First flatten `F` to the form `F = F1 &...& FN`.
+* For each `Fi`
+ * If `Fi` is a function type `T -> U`, try unifying `X` with `T`. If unification succeeds, return `U`.
+ * If `Fi` is a free type, unify it with `X -> Y` (for fresh free type `Y`) and return `Y`.
+ * Otherwise, report an error.
+* If we got here, overload resolution failed, so we bail and return the return type of one of the overloads.
+
+For example:
+
+* if `F` is `((number? -> string?) & (string? -> number?)` and `X` is free, then `X` is unified with `number?` and the result type is `string?`
+* if `F` is `((number? -> string?) & (string? -> number?)` and `X` is `(number | string)`, then an error is reported and the result type is `string?`
+* if `F` is `((number? -> string?) & (string? -> number?)` and `X` is `nil`, then the result type is `string?`
+
+The order of constraints for this algorithm matters, for example:
+
+```lua
+  type T = { f : (number -> string) & (string -> number) }
+  local function useT(x : T) x.f("hi") end
+  local function (x)
+    x.f(0)  -- this unifies the type of x.f with number -> Y
+    useT(x) -- this does not typecheck because T gives a more precise type for f
+  end
+```
+
+but swapping the order of constraints:
+```lua
+  type T = { f : (number -> string) & (string -> number) }
+  local function useT(x : T) x.f("hi") end
+  local function (x)
+    useT(x) -- unifies the type of x with T
+    x.f(0)  -- this is fine
+  end
+```
+
+This is okay for greedy type inference, where the order of constraints
+is important, but not great for local type inference, which is meant
+to be independent of constraint order.
+
+## Design
+
+In this proposal, we delay producing a type for function applications until the function and argument types are concrete.
+
+We do this by introducing types `src F` and `tgt F(X)` (syntax to be bikeshedded, but
+hopefully it will never be seen by creators!), standing for the parameter and
+result types of applying a function of type `F` to an argument of type `X`.
+
+The algorithm for function overload resolution is now trivial: the type of `f(x)` is just `tgt F(X)`, and this introduces a constraint `X <: src(F)`.
+
+The extra work is now in type normalization, which removes uses of `src` and `tgt` during quantification:
+
+```
+  src (T -> U) is  T 
+  src (T & U)  is  src T | src U
+  src (T | U)  is  src T & src U
+  src T        is  src F when T is a table with a metatable with a __call__ property of type F
+  src T        is  none in any other case
+
+  tgt (T -> U) (V) is  U    when (T&V) is inhabited
+  tgt (T & U) (V)  is  tgt T(V) & tgt U(V)
+  tgt (T | U) (V)  is  tgt T(V) | tgt U(V)
+  tgt T (V)        is  tgt F(V) when T is a table with a metatable with a __call__ property of type F
+  tgt top (T)      is  top
+  tgt T (V)        is  none in any other case
+```
+
+For example:
+
+```
+  src((number? -> string?) & (string? -> number?))                    is  (number | string)?
+  tgt((number? -> string?) & (string? -> number?)) (number)           is  string?
+  tgt((number? -> string?) & (string? -> number?)) (number | string)  is  (number | string)?
+  tgt((number? -> string?) & (string? -> number?)) (nil)              is  nil
+```
+
+(This is a variant of the algorithm used in the Luau prototype.)
+
+The real thing would need scaled up to type packs and generic functions.
+
+## Drawbacks
+
+Anything we do about this will be a breaking change.
+
+Scaling this up to type packs probably involves intersection and union on packs.
+
+There may be other devils in the details.
+
+We need to ensure `src` and `tgt` do not leak.
+
+We need to make sure that normalization doesn't lose valuable autocomplete information.
+
+We need an algorithm for "is a type inhabited".
+
+## Alternatives
+
+We could try applying fixes to the current algorithm, and live with constraint order mattering.
+
+We could adopt the Flow solution, which adds "if this then that" constraints, but that makes it very likely we will need to introduce additional backtracking.