Reland "Reland "[ios] Improve build/toolchain/ios/swiftc.py script""
This is a reland of commit e628fca793dea9493483753d202c5ca54a52cb9d
With https://gn-review.googlesource.com/16860, the ninja build action
that invokes the swiftc.py script has `restat = 1`. This causes ninja
to support the optimisation that output are not changed if not needed
(e.g. if input was just reformated, the compilation was not required).
This ensures that "no-op" builds are not considered dirty, and allow
to re-enable the FileWriter optimisation (to allow swiftc to see that
the compilation is not needed either, when possible).
Checked locally that this fixes the "no-op" build considered dirty
failure that was reproducible at 100% rate with previous version of
gn (without `restat = 1`).
Original change's description:
> Reland "[ios] Improve build/toolchain/ios/swiftc.py script"
>
> This is a reland of commit 555b7f43f59a6291b935c1dd8e99fc6ce77d183e
>
> Fixes the following issues that lead to reverting the original CL:
>
> 1. The compilation was sometimes considered dirty after an incremental
> build due to optimizations in both the script (FileWriter) and in
> swiftc itself where output file are not overwritten if identical.
>
> This was fixed by removing the use of FileWriter from the script and
> by touching all the outputs of the compilation (thus the file should
> now have newer timestamp than the source).
>
> 2. The compilation for catalyst environment was broken because swiftc
> expects different command-line argument for that environment (i.e.
> -isystem is unsupported, -Xcc ... is also unsupported).
>
> The script is modified to check whether the target triple
> corresponds to the catalyst environment (i.e. matches *-macabi)
> and constructs an acceptable command-line.
>
> 3. As `gn check ...` did not know about the Objective-C generated
> header for Swift modules, some dependencies were missing which
> also caused dirty build after an incremental build. This was
> fixed by https://crrev.com/c/5348787.
>
> Original change's description:
> > [ios] Improve build/toolchain/ios/swiftc.py script
> >
> > Change the script build/toolchain/ios/swiftc.py to configure the
> > swift compiler to generate $input.swiftconstvalues files (needed
> > for Intents API in swift).
> >
> > Improve the code to put all the derived data in one directory,
> > which allow removing the need for -pch-output-dir $dir from the
> > invocation of the script and its list of outputs (should avoid
> > the compilation error where $dir is created as a file instead of
> > a directory).
> >
> > Change toolchain.gni to pass all the tool versions as environment
> > variables instead of command-line arguments (reduce the length of
> > the command-line and remove the need to parse those arguments in
> > the script).
> >
> > Allow keeping the derived data dir between invocations of the
> > script (which may improve the build performance as the swiftc
> > binary will not have to rebuild the module cache each time).
> >
> > Bug: 326059372
> > Change-Id: I5e42676eb917fe2d355383aaa39a60d5674816fb
> > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5319176
> > Commit-Queue: Sylvain Defresne <sdefresne@chromium.org>
> > Reviewed-by: Federica Germinario <fedegermi@google.com>
> > Cr-Commit-Position: refs/heads/main@{#1268444}
>
> Bug: 326059372
> Cq-Include-Trybots: luci.chromium.try:ios-catalyst
> Change-Id: I94b9783441b2f92dc3e6caf8bb6a3522bc0cb419
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5349118
> Reviewed-by: Federica Germinario <fedegermi@google.com>
> Commit-Queue: Sylvain Defresne <sdefresne@chromium.org>
> Cr-Commit-Position: refs/heads/main@{#1269482}
Bug: 326059372
Change-Id: Ibb2d8aa5b169ec4f0fb2c5b413ba5b54eb50dde1
Cq-Include-Trybots: luci.chromium.try:ios-catalyst
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5352649
Reviewed-by: Federica Germinario <fedegermi@google.com>
Commit-Queue: Sylvain Defresne <sdefresne@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1272108}
diff --git a/build/toolchain/apple/toolchain.gni b/build/toolchain/apple/toolchain.gni
index 8fd4293..37360d2 100644
--- a/build/toolchain/apple/toolchain.gni
+++ b/build/toolchain/apple/toolchain.gni
@@ -36,6 +36,11 @@
# for official builds, and "false" for all other builds.
swift_whole_module_optimization = -1
+ # If true, the intermediate build products of swift module compilation will
+ # be kept after the invocation of the swiftc compiler. Otherwise they will
+ # deleted between each invocation.
+ swift_keep_intermediate_files = false
+
# If unspecified, will use the toolchain downloaded via deps.
swift_toolchain_path = -1
}
@@ -76,16 +81,17 @@
# but it will cause edges to be marked as dirty if the ninja files are
# regenerated. See https://crbug.com/619083 for details. A proper fix
# would be to have inputs to tools (https://crbug.com/621119).
-tool_versions =
- exec_script("get_tool_mtime.py",
- rebase_path([
- "//build/toolchain/apple/filter_libtool.py",
- "//build/toolchain/apple/linker_driver.py",
- "//build/toolchain/ios/compile_xcassets.py",
- "//build/toolchain/ios/swiftc.py",
- ],
- root_build_dir),
- "trim scope")
+tool_versions = exec_script(
+ "get_tool_mtime.py",
+ rebase_path([
+ "//build/toolchain/apple/filter_libtool.py",
+ "//build/toolchain/apple/linker_driver.py",
+ "//build/toolchain/ios/compile_xcassets.py",
+ "//build/toolchain/ios/const_extract_protocols.json",
+ "//build/toolchain/ios/swiftc.py",
+ ],
+ root_build_dir),
+ "trim scope")
# Shared toolchain definition. Invocations should set current_os to set the
# build args in this definition. This is titled "single_apple_toolchain"
@@ -645,73 +651,64 @@
depfile = "{{target_out_dir}}/{{module_name}}.d"
depsformat = "gcc"
- outputs = [
- # The module needs to be the first output listed. The blank line after
- # the module is required to prevent `gn format` from changing the file
- # order.
- "{{target_gen_dir}}/{{module_name}}.swiftmodule",
+ _header_path = "{{target_gen_dir}}/{{target_output_name}}.h"
+ _output_dir = "{{target_out_dir}}/{{label_name}}"
- "{{target_gen_dir}}/{{target_output_name}}.h",
- "{{target_gen_dir}}/{{module_name}}.swiftdoc",
- "{{target_gen_dir}}/{{module_name}}.swiftsourceinfo",
+ outputs = [
+ _header_path,
+ "$_output_dir/{{module_name}}-OutputFileMap.json",
+ "$_output_dir/{{module_name}}.SwiftFileList",
+ "$_output_dir/{{module_name}}.abi.json",
+ "$_output_dir/{{module_name}}.d",
+ "$_output_dir/{{module_name}}.dia",
+ "$_output_dir/{{module_name}}.swiftdoc",
+ "$_output_dir/{{module_name}}.swiftmodule",
+ "$_output_dir/{{module_name}}.swiftsourceinfo",
]
+ partial_outputs = [ "$_output_dir/{{source_name_part}}.o" ]
+
+ # The list of outputs and partial_outputs change whether the whole
+ # module optimization is enabled or not.
+ if (swift_whole_module_optimization) {
+ outputs += [
+ "$_output_dir/{{module_name}}.swiftconstvalues",
+ "$_output_dir/{{module_name}}.swiftdeps",
+ ]
+ } else {
+ outputs += [ "$_output_dir/{{module_name}}.priors" ]
+ partial_outputs += [
+ "$_output_dir/{{source_name_part}}.d",
+ "$_output_dir/{{source_name_part}}.dia",
+ "$_output_dir/{{source_name_part}}.swiftdeps",
+ "$_output_dir/{{source_name_part}}.swiftconstvalues",
+ ]
+ }
+
+ # If configured to keep the intermediate build files, pass the flag
+ # to the script and inform gn of the stamp file only (as the other
+ # files have names that cannot be predicted without invoking swiftc).
+ if (swift_keep_intermediate_files) {
+ _derived_data_dir = "$_output_dir/DerivedData"
+ outputs += [ "$_derived_data_dir/{{module_name}}.stamp" ]
+ }
+
# Additional flags passed to the wrapper script but that are only
# set conditionally.
_extra_flags = ""
- if (swift_whole_module_optimization) {
- _extra_flags += " -whole-module-optimization"
- _objects_dir = "{{target_out_dir}}"
-
- outputs += [ "$_objects_dir/{{module_name}}.o" ]
- } else {
- _objects_dir = "{{target_out_dir}}/{{label_name}}"
-
- partial_outputs = [ "$_objects_dir/{{source_name_part}}.o" ]
- }
-
- _env_vars = "TOOL_VERSION=${tool_versions.swiftc}"
- if (invoker.sdk_developer_dir != "") {
- _env_vars += " DEVELOPER_DIR=${toolchain_args.sdk_developer_dir}"
- }
-
- # Starting with version 5.6, the Swift compiler will always
- # generates precompiled headers. In anterior version, it was
- # used when bridging headers and whole module optimisation
- # where enabled, and it could be disabled with the parameter
- # `-disable-bridging-pch`.
- #
- # The precompiled headers are binary files (i.e. they are not
- # regular Objective-C header files and cannot be loaded as such).
- #
- # There is an hidden requirements that the compiler needs to
- # be told where to save those .pch files (via the parameter
- # `-pch-output-dir $dir`). If this parameter is not passed, the
- # compiler will silently write them at an incorrect location,
- # leading later pass to try to load those .pch files as either
- # regular header files (.h) or object files (.o) and causing
- # to compilation failures.
- #
- # List the directory where the precompiled header is generated
- # as an output, but do not list the .pch file itself. This is
- # because the names includes two hashes (one corresponding to
- # the compiler revision, and the other probably derived from
- # the module itself) that are difficult to generate.
- #
- # Still we want to avoid creating a directory that has the same
- # name as a file generated by another rule, so explicitly list
- # the directory in `outputs` so that gn can warn it is conflicts
- # with another output file.
-
- _pch_output_dir = "{{target_out_dir}}/{{module_name}}:pch/"
- outputs += [ "$_pch_output_dir/{{module_name}}.stamp" ]
+ # Environment variables passed to the wrapper script. Considered
+ # part of the command-line by ninja (and thus cause the build to
+ # be considered dirty if they change) without having to be parsed
+ # by the script.
+ _env_vars = "TOOL_VERSION=${tool_versions.swiftc} " +
+ "JSON_VERSION=${tool_versions.const_extract_protocols}"
# Include the version of the compiler on the command-line. This causes
# `ninja` to consider all the compilation output to be dirty when the
# version changes.
if (defined(swiftc_version)) {
- _extra_flags += " -swiftc-version $swiftc_version"
+ _env_vars += " SWIFTC_VERSION=$swiftc_version"
}
# Include the version of Xcode on the command-line (if specified via
@@ -724,7 +721,11 @@
# changes, they may encode dependency on now non-existing frameworks
# causing linker failures ultimately.
if (defined(toolchain_args.xcode_build)) {
- _extra_flags += " -xcode-version ${toolchain_args.xcode_build}"
+ _env_vars += " XCODE_VERSION=${toolchain_args.xcode_build}"
+ }
+
+ if (invoker.sdk_developer_dir != "") {
+ _env_vars += " DEVELOPER_DIR=${toolchain_args.sdk_developer_dir}"
}
if (swift_toolchain_path != "") {
@@ -732,6 +733,14 @@
rebase_path(swift_toolchain_path, root_build_dir)
}
+ if (swift_whole_module_optimization) {
+ _extra_flags += " -whole-module-optimization"
+ }
+
+ if (swift_keep_intermediate_files) {
+ _extra_flags += " -derived-data-dir $_derived_data_dir"
+ }
+
# The Swift compiler assumes that the generated header will be used by
# Objective-C code compiled with module support enabled (-fmodules).
#
@@ -758,16 +767,20 @@
# around those two issues.
_extra_flags += " -fix-module-imports"
+ _src_dir = rebase_path("//", root_build_dir)
+ _const_gather_protocols_file =
+ rebase_path("//build/toolchain/ios/const_extract_protocols.json",
+ root_build_dir)
+
command =
"$_env_vars $python_path $_tool -module-name {{module_name}} " +
- "-root-dir " + rebase_path("//", root_build_dir) + " " +
- "-object-dir $_objects_dir -pch-output-dir $_pch_output_dir " +
- "-module-path {{target_gen_dir}}/{{module_name}}.swiftmodule " +
- "-header-path {{target_gen_dir}}/{{target_output_name}}.h " +
- "-depfile {{target_out_dir}}/{{module_name}}.d " +
- "-bridge-header {{bridge_header}} $_extra_flags " +
- "{{swiftflags}} {{include_dirs}} {{module_dirs}} {{inputs}}"
- description = "SWIFT {{output}}"
+ "-header-path $_header_path -target-out-dir $_output_dir " +
+ "-const-gather-protocols-file $_const_gather_protocols_file " +
+ "-depfile-path $depfile -src-dir $_src_dir -bridge-header " +
+ "{{bridge_header}} {{include_dirs}} {{module_dirs}} " +
+ "{{swiftflags}} {{inputs}}$_extra_flags"
+
+ description = "SWIFT $_output_dir/{{module_name}}.swiftmodule"
}
}
diff --git a/build/toolchain/ios/const_extract_protocols.json b/build/toolchain/ios/const_extract_protocols.json
new file mode 100644
index 0000000..fffab8ac
--- /dev/null
+++ b/build/toolchain/ios/const_extract_protocols.json
@@ -0,0 +1 @@
+["AppIntent","EntityQuery","AppEntity","TransientEntity","AppEnum","AppShortcutProviding","AppShortcutsProvider","AnyResolverProviding","AppIntentsPackage","DynamicOptionsProvider"]
\ No newline at end of file
diff --git a/build/toolchain/ios/swiftc.py b/build/toolchain/ios/swiftc.py
index b8f7b5d4..85d2728 100644
--- a/build/toolchain/ios/swiftc.py
+++ b/build/toolchain/ios/swiftc.py
@@ -3,13 +3,275 @@
# found in the LICENSE file.
import argparse
+import collections
+import contextlib
+import hashlib
+import io
import json
+import multiprocessing
import os
+import shutil
import subprocess
import sys
import tempfile
+class ArgumentForwarder(object):
+ """Class used to abstract forwarding arguments from to the swiftc compiler.
+
+ Arguments:
+ - arg_name: string corresponding to the argument to pass to the compiler
+ - arg_join: function taking the compiler name and returning whether the
+ argument value is attached to the argument or separated
+ - to_swift: function taking the argument value and returning whether it
+ must be passed to the swift compiler
+ - to_clang: function taking the argument value and returning whether it
+ must be passed to the clang compiler
+ """
+
+ def __init__(self, arg_name, arg_join, to_swift, to_clang):
+ self._arg_name = arg_name
+ self._arg_join = arg_join
+ self._to_swift = to_swift
+ self._to_clang = to_clang
+
+ def forward(self, swiftc_args, values, target_triple):
+ if not values:
+ return
+
+ is_catalyst = target_triple.endswith('macabi')
+ for value in values:
+ if self._to_swift(value):
+ if self._arg_join('swift'):
+ swiftc_args.append(f'{self._arg_name}{value}')
+ else:
+ swiftc_args.append(self._arg_name)
+ swiftc_args.append(value)
+
+ if self._to_clang(value) and not is_catalyst:
+ if self._arg_join('clang'):
+ swiftc_args.append('-Xcc')
+ swiftc_args.append(f'{self._arg_name}{value}')
+ else:
+ swiftc_args.append('-Xcc')
+ swiftc_args.append(self._arg_name)
+ swiftc_args.append('-Xcc')
+ swiftc_args.append(value)
+
+
+class IncludeArgumentForwarder(ArgumentForwarder):
+ """Argument forwarder for -I and -isystem."""
+
+ def __init__(self, arg_name):
+ ArgumentForwarder.__init__(self,
+ arg_name,
+ arg_join=lambda _: len(arg_name) == 1,
+ to_swift=lambda _: arg_name != '-isystem',
+ to_clang=lambda _: True)
+
+
+class FrameworkArgumentForwarder(ArgumentForwarder):
+ """Argument forwarder for -F and -Fsystem."""
+
+ def __init__(self, arg_name):
+ ArgumentForwarder.__init__(self,
+ arg_name,
+ arg_join=lambda _: len(arg_name) == 1,
+ to_swift=lambda _: True,
+ to_clang=lambda _: True)
+
+
+class DefineArgumentForwarder(ArgumentForwarder):
+ """Argument forwarder for -D."""
+
+ def __init__(self, arg_name):
+ ArgumentForwarder.__init__(self,
+ arg_name,
+ arg_join=lambda _: _ == 'clang',
+ to_swift=lambda _: '=' not in _,
+ to_clang=lambda _: True)
+
+
+# Dictionary mapping argument names to their ArgumentForwarder.
+ARGUMENT_FORWARDER_FOR_ATTR = (
+ ('include_dirs', IncludeArgumentForwarder('-I')),
+ ('system_include_dirs', IncludeArgumentForwarder('-isystem')),
+ ('framework_dirs', FrameworkArgumentForwarder('-F')),
+ ('system_framework_dirs', FrameworkArgumentForwarder('-Fsystem')),
+ ('defines', DefineArgumentForwarder('-D')),
+)
+
+
+class FileWriter(contextlib.AbstractContextManager):
+ """
+ FileWriter is a file-like object that only write data to disk if changed.
+
+ This object implements the context manager protocols and thus can be used
+ in a with-clause. The data is written to disk when the context is exited,
+ and only if the content is different from current file content.
+
+ with FileWriter(path) as stream:
+ stream.write('...')
+
+ If the with-clause ends with an exception, no data is written to the disk
+ and any existing file is left untouched.
+ """
+
+ def __init__(self, filepath, encoding='utf8'):
+ self._stringio = io.StringIO()
+ self._filepath = filepath
+ self._encoding = encoding
+
+ def __exit__(self, exc_type, exc_value, traceback):
+ if exc_type or exc_value or traceback:
+ return
+
+ new_content = self._stringio.getvalue()
+ if os.path.exists(self._filepath):
+ with open(self._filepath, encoding=self._encoding) as stream:
+ old_content = stream.read()
+
+ if old_content == new_content:
+ return
+
+ with open(self._filepath, 'w', encoding=self._encoding) as stream:
+ stream.write(new_content)
+
+ def write(self, data):
+ self._stringio.write(data)
+
+
+@contextlib.contextmanager
+def existing_directory(path):
+ """Returns a context manager wrapping an existing directory."""
+ yield path
+
+
+def create_stamp_file(path):
+ """Writes an empty stamp file at path."""
+ with FileWriter(path) as stream:
+ stream.write('')
+
+
+def create_build_cache_dir(args, build_signature):
+ """Creates the build cache directory according to `args`.
+
+ This function returns an object that implements the context manager
+ protocol and thus can be used in a with-clause. If -derived-data-dir
+ argument is not used, the returned directory is a temporary directory
+ that will be deleted when the with-clause is exited.
+ """
+ if not args.derived_data_dir:
+ return tempfile.TemporaryDirectory()
+
+ # The derived data cache can be quite large, so delete any obsolete
+ # files or directories.
+ stamp_name = f'{args.module_name}.stamp'
+ if os.path.isdir(args.derived_data_dir):
+ for name in os.listdir(args.derived_data_dir):
+ if name not in (build_signature, stamp_name):
+ path = os.path.join(args.derived_data_dir, name)
+ if os.path.isdir(path):
+ shutil.rmtree(path)
+ else:
+ os.unlink(path)
+
+ ensure_directory(args.derived_data_dir)
+ create_stamp_file(os.path.join(args.derived_data_dir, stamp_name))
+
+ return existing_directory(
+ ensure_directory(os.path.join(args.derived_data_dir, build_signature)))
+
+
+def ensure_directory(path):
+ """Creates directory at `path` if it does not exists."""
+ if not os.path.isdir(path):
+ os.makedirs(path)
+ return path
+
+
+def build_signature(env, args):
+ """Generates the build signature from `env` and `args`.
+
+ This allow re-using the derived data dir between builds while still
+ forcing the data to be recreated from scratch in case of significant
+ changes to the build settings (different arguments or tool versions).
+ """
+ m = hashlib.sha1()
+ for key in sorted(env):
+ if key.endswith('_VERSION') or key == 'DEVELOPER_DIR':
+ m.update(f'{key}={env[key]}'.encode('utf8'))
+ for i, arg in enumerate(args):
+ m.update(f'{i}={arg}'.encode('utf8'))
+ return m.hexdigest()
+
+
+def generate_source_output_file_map_fragment(args, filename):
+ """Generates source OutputFileMap.json fragment according to `args`.
+
+ Create the fragment for a single .swift source file for OutputFileMap.
+ The output depends on whether -whole-module-optimization argument is
+ used or not.
+ """
+ assert os.path.splitext(filename)[1] == '.swift', filename
+ basename = os.path.splitext(os.path.basename(filename))[0]
+ rel_name = os.path.join(args.target_out_dir, basename)
+ out_name = rel_name
+
+ fragment = {
+ 'index-unit-output-path': f'/{rel_name}.o',
+ 'object': f'{out_name}.o',
+ }
+
+ if not args.whole_module_optimization:
+ fragment.update({
+ 'const-values': f'{out_name}.swiftconstvalues',
+ 'dependencies': f'{out_name}.d',
+ 'diagnostics': f'{out_name}.dia',
+ 'swift-dependencies': f'{out_name}.swiftdeps',
+ })
+
+ return fragment
+
+
+def generate_module_output_file_map_fragment(args):
+ """Generates module OutputFileMap.json fragment according to `args`.
+
+ Create the fragment for the module itself for OutputFileMap. The output
+ depends on whether -whole-module-optimization argument is used or not.
+ """
+ out_name = os.path.join(args.target_out_dir, args.module_name)
+
+ if args.whole_module_optimization:
+ fragment = {
+ 'const-values': f'{out_name}.swiftconstvalues',
+ 'dependencies': f'{out_name}.d',
+ 'diagnostics': f'{out_name}.dia',
+ 'swift-dependencies': f'{out_name}.swiftdeps',
+ }
+ else:
+ fragment = {
+ 'emit-module-dependencies': f'{out_name}.d',
+ 'emit-module-diagnostics': f'{out_name}.dia',
+ 'swift-dependencies': f'{out_name}.swiftdeps',
+ }
+
+ return fragment
+
+
+def generate_output_file_map(args):
+ """Generates OutputFileMap.json according to `args`.
+
+ Returns the mapping as a python dictionary that can be serialized to
+ disk as JSON.
+ """
+ output_file_map = {'': generate_module_output_file_map_fragment(args)}
+ for filename in args.sources:
+ fragment = generate_source_output_file_map_fragment(args, filename)
+ output_file_map[filename] = fragment
+ return output_file_map
+
+
def fix_module_imports(header_path, output_path):
"""Convert modules import to work without -fmodules support.
@@ -25,308 +287,324 @@
"""
header_contents = []
- with open(header_path, 'r') as header_file:
+ with open(header_path, 'r', encoding='utf8') as header_file:
+
+ imports_section = None
for line in header_file:
- if line == '#if __has_feature(objc_modules)\n':
- header_contents.append('#if 1 // #if __has_feature(objc_modules)\n')
- nesting_level = 1
- for line in header_file:
- if line == '#endif\n':
- nesting_level -= 1
- elif line.startswith('@import'):
- name = line.split()[1].split(';')[0]
- if name != 'ObjectiveC':
- header_contents.append(f'#import <{name}/{name}.h> ')
- header_contents.append('// ')
- elif line.startswith('#if'):
- nesting_level += 1
+ if line.startswith('#if __has_feature(objc_modules)'):
+ assert imports_section is None
+ imports_section = (len(header_contents) + 1, 1)
+ elif imports_section:
+ section_start, nesting_level = imports_section
+ if line.startswith('#if'):
+ imports_section = (section_start, nesting_level + 1)
+ elif line.startswith('#endif'):
+ if nesting_level > 1:
+ imports_section = (section_start, nesting_level - 1)
+ else:
+ imports_section = None
+ section_end = len(header_contents)
+ header_contents.append('#else\n')
+ for index in range(section_start, section_end):
+ l = header_contents[index]
+ if l.startswith('@import'):
+ name = l.split()[1].split(';')[0]
+ if name != 'ObjectiveC':
+ header_contents.append(f'#import <{name}/{name}.h>\n')
+ else:
+ header_contents.append(l)
+ header_contents.append(line)
- header_contents.append(line)
- if nesting_level == 0:
- break
- else:
- header_contents.append(line)
-
- with open(output_path, 'w') as header_file:
+ with FileWriter(output_path) as header_file:
for line in header_contents:
header_file.write(line)
-def compile_module(module, sources, settings, extras, tmpdir):
- """Compile `module` from `sources` using `settings`."""
- output_file_map = {}
- if settings.whole_module_optimization:
- output_file_map[''] = {
- 'object': os.path.join(settings.object_dir, module + '.o'),
- 'dependencies': os.path.join(tmpdir, module + '.d'),
- }
- else:
- for source in sources:
- name, _ = os.path.splitext(os.path.basename(source))
- output_file_map[source] = {
- 'object': os.path.join(settings.object_dir, name + '.o'),
- 'dependencies': os.path.join(tmpdir, name + '.d'),
- }
+def invoke_swift_compiler(args, extras_args, build_cache_dir, output_file_map):
+ """Invokes Swift compiler to compile module according to `args`.
- for key in ('module_path', 'header_path', 'depfile'):
- path = getattr(settings, key)
- if os.path.exists(path):
- os.unlink(path)
- if key == 'module_path':
- for ext in '.swiftdoc', '.swiftsourceinfo':
- path = os.path.splitext(getattr(settings, key))[0] + ext
- if os.path.exists(path):
- os.unlink(path)
- directory = os.path.dirname(path)
- if not os.path.exists(directory):
- os.makedirs(directory)
+ The `build_cache_dir` and `output_file_map` should be path to existing
+ directory to use for writing intermediate build artifact (optionally
+ a temporary directory) and path to $module-OutputFileMap.json file that
+ lists the outputs to generate for the module and each source file.
- if not os.path.exists(settings.object_dir):
- os.makedirs(settings.object_dir)
+ If -fix-module-imports argument is passed, the generated header for the
+ module is written to a temporary location and then modified to replace
+ @import by corresponding #import.
+ """
- if not os.path.exists(settings.pch_output_dir):
- os.makedirs(settings.pch_output_dir)
+ # Write the $module.SwiftFileList file.
+ swift_file_list_path = os.path.join(args.target_out_dir,
+ f'{args.module_name}.SwiftFileList')
- stamp = os.path.join(settings.pch_output_dir, f'{module}.stamp')
- if os.path.exists(stamp):
- os.unlink(stamp)
- with open(stamp, 'w') as stamp_file:
- stamp_file.write('')
+ with FileWriter(swift_file_list_path) as stream:
+ for filename in sorted(args.sources):
+ stream.write(filename)
+ stream.write('\n')
- for key in output_file_map:
- path = output_file_map[key]['object']
- if os.path.exists(path):
- os.unlink(path)
+ header_path = args.header_path
+ if args.fix_module_imports:
+ header_path = os.path.join(build_cache_dir, os.path.basename(header_path))
- output_file_map.setdefault('', {})['swift-dependencies'] = \
- os.path.join(tmpdir, module + '.swift.d')
-
- output_file_map_path = os.path.join(tmpdir, module + '.json')
- with open(output_file_map_path, 'w') as output_file_map_file:
- output_file_map_file.write(json.dumps(output_file_map))
- output_file_map_file.flush()
-
- extra_args = []
- if settings.file_compilation_dir:
- extra_args.extend([
- '-file-compilation-dir',
- settings.file_compilation_dir,
- ])
-
- if settings.bridge_header:
- extra_args.extend([
- '-import-objc-header',
- os.path.abspath(settings.bridge_header),
- ])
-
- if settings.whole_module_optimization:
- extra_args.append('-whole-module-optimization')
-
- if settings.target:
- extra_args.extend([
- '-target',
- settings.target,
- ])
-
- if settings.sdk:
- extra_args.extend([
- '-sdk',
- os.path.abspath(settings.sdk),
- ])
-
- if settings.swift_version:
- extra_args.extend([
- '-swift-version',
- settings.swift_version,
- ])
-
- if settings.include_dirs:
- for include_dir in settings.include_dirs:
- extra_args.append('-I' + include_dir)
-
- if settings.system_include_dirs:
- for system_include_dir in settings.system_include_dirs:
- extra_args.extend(['-Xcc', '-isystem', '-Xcc', system_include_dir])
-
- if settings.framework_dirs:
- for framework_dir in settings.framework_dirs:
- extra_args.extend([
- '-F',
- framework_dir,
- ])
-
- if settings.system_framework_dirs:
- for system_framework_dir in settings.system_framework_dirs:
- extra_args.extend([
- '-F',
- system_framework_dir,
- ])
-
- if settings.enable_cxx_interop:
- extra_args.extend([
- '-Xfrontend',
- '-enable-cxx-interop',
- ])
-
- # The swiftc compiler uses a global module cache that is not robust against
- # changes in the sub-modules nor against corruption (see crbug.com/1358073).
- # Force the compiler to store the module cache in a sub-directory of `tmpdir`
- # to ensure a pristine module cache is used for every compiler invocation.
- module_cache_path = os.path.join(tmpdir, settings.swiftc_version,
- 'ModuleCache')
-
- # If the generated header is post-processed, generate it to a temporary
- # location (to avoid having the file appear to suddenly change).
- if settings.fix_module_imports:
- header_path = os.path.join(tmpdir, f'{module}.h')
- else:
- header_path = settings.header_path
-
- process = subprocess.Popen([
- settings.swift_toolchain_path + '/usr/bin/swiftc',
+ swiftc_args = [
'-parse-as-library',
'-module-name',
- module,
- '-module-cache-path',
- module_cache_path,
- '-emit-object',
+ args.module_name,
+ f'@{swift_file_list_path}',
+ '-sdk',
+ args.sdk_path,
+ '-target',
+ args.target_triple,
+ '-swift-version',
+ args.swift_version,
+ '-c',
+ '-output-file-map',
+ output_file_map,
+ '-save-temps',
+ '-no-color-diagnostics',
+ '-serialize-diagnostics',
'-emit-dependencies',
'-emit-module',
'-emit-module-path',
- settings.module_path,
+ os.path.join(args.target_out_dir, f'{args.module_name}.swiftmodule'),
'-emit-objc-header',
'-emit-objc-header-path',
header_path,
- '-output-file-map',
- output_file_map_path,
+ '-working-directory',
+ os.getcwd(),
+ '-index-store-path',
+ ensure_directory(os.path.join(build_cache_dir, 'Index.noindex')),
+ '-module-cache-path',
+ ensure_directory(os.path.join(build_cache_dir, 'ModuleCache.noindex')),
'-pch-output-dir',
- os.path.abspath(settings.pch_output_dir),
- ] + extra_args + extras + sources)
+ ensure_directory(os.path.join(build_cache_dir, 'PrecompiledHeaders')),
+ ]
+ # Handle optional -bridge-header flag.
+ if args.bridge_header:
+ swiftc_args.extend(('-import-objc-header', args.bridge_header))
+
+ # Handle swift const values extraction.
+ swiftc_args.extend(['-emit-const-values'])
+ swiftc_args.extend([
+ '-Xfrontend',
+ '-const-gather-protocols-file',
+ '-Xfrontend',
+ args.const_gather_protocols_file,
+ ])
+
+ # Handle -I, -F, -isystem, -Fsystem and -D arguments.
+ for (attr_name, forwarder) in ARGUMENT_FORWARDER_FOR_ATTR:
+ forwarder.forward(swiftc_args, getattr(args, attr_name), args.target_triple)
+
+ # Handle -whole-module-optimization flag.
+ num_threads = max(1, multiprocessing.cpu_count() // 2)
+ if args.whole_module_optimization:
+ swiftc_args.extend([
+ '-whole-module-optimization',
+ '-no-emit-module-separately-wmo',
+ '-num-threads',
+ f'{num_threads}',
+ ])
+ else:
+ swiftc_args.extend([
+ '-enable-batch-mode',
+ '-incremental',
+ '-experimental-emit-module-separately',
+ '-disable-cmo',
+ f'-j{num_threads}',
+ ])
+
+ swift_toolchain_path = args.swift_toolchain_path
+ if not swift_toolchain_path:
+ swift_toolchain_path = os.path.join(os.path.dirname(args.sdk_path),
+ 'XcodeDefault.xctoolchain')
+ if not os.path.isdir(swift_toolchain_path):
+ swift_toolchain_path = ''
+
+ command = [f'{swift_toolchain_path}/usr/bin/swiftc'] + swiftc_args
+ if extras_args:
+ command.extend(extras_args)
+
+ process = subprocess.Popen(command)
process.communicate()
+
if process.returncode:
sys.exit(process.returncode)
- if settings.fix_module_imports:
- fix_module_imports(header_path, settings.header_path)
+ if args.fix_module_imports:
+ fix_module_imports(header_path, args.header_path)
- # The swiftc compiler generates depfile that uses absolute paths, but
- # ninja requires paths in depfiles to be identical to paths used in
- # the build.ninja files.
- #
- # Since gn generates paths relative to the build directory for all paths
- # below the repository checkout, we need to convert those to relative
- # paths.
- #
- # See https://crbug.com/1287114 for build failure that happen when the
- # paths in the depfile are kept absolute.
- out_dir = os.getcwd() + os.path.sep
- src_dir = os.path.abspath(settings.root_dir) + os.path.sep
- depfile_content = dict()
- for key in output_file_map:
+def generate_depfile(args, output_file_map):
+ """Generates compilation depfile according to `args`.
- # When whole module optimisation is disabled, there will be an entry
- # with an empty string as the key and only ('swift-dependencies') as
- # keys in the value dictionary. This is expected, so skip entry that
- # do not include 'dependencies' in their keys.
- depencency_file_path = output_file_map[key].get('dependencies')
- if not depencency_file_path:
- continue
+ Parses all intermediate depfile generated by the Swift compiler and
+ replaces absolute path by relative paths (since ninja compares paths
+ as strings and does not resolve relative paths to absolute).
- for line in open(depencency_file_path):
- output, inputs = line.split(' : ', 2)
- _, ext = os.path.splitext(output)
- if ext == '.o':
- key = output
- else:
- key = os.path.splitext(settings.module_path)[0] + ext
- if key not in depfile_content:
- depfile_content[key] = set()
- for path in inputs.split():
- if path.startswith(src_dir) or path.startswith(out_dir):
- path = os.path.relpath(path, out_dir)
- depfile_content[key].add(path)
+ Converts path to the SDK and toolchain files to the sdk/xcode_link
+ symlinks if possible and available.
+ """
+ xcode_paths = {}
+ if os.path.islink(args.sdk_path):
+ xcode_links = os.path.dirname(args.sdk_path)
+ for link_name in os.listdir(xcode_links):
+ link_path = os.path.join(xcode_links, link_name)
+ if os.path.islink(link_path):
+ xcode_paths[os.path.realpath(link_path) + os.sep] = link_path + os.sep
- with open(settings.depfile, 'w') as depfile:
- keys = sorted(depfile_content.keys())
- for key in sorted(keys):
- depfile.write('%s : %s\n' % (key, ' '.join(sorted(depfile_content[key]))))
+ out_dir = os.getcwd() + os.sep
+ src_dir = os.path.abspath(args.src_dir) + os.path.sep
+
+ depfile_content = collections.defaultdict(set)
+ for value in output_file_map.values():
+ partial_depfile_path = value.get('dependencies', None)
+ if partial_depfile_path:
+ with open(partial_depfile_path, encoding='utf8') as stream:
+ for line in stream:
+ output, inputs = line.split(' : ', 2)
+ output = os.path.relpath(output, out_dir)
+
+ for path in inputs.split():
+ for xcode_path in xcode_paths:
+ if path.startswith(xcode_path):
+ path = xcode_paths[xcode_path] + path[len(xcode_path):]
+ if path.startswith(src_dir) or path.startswith(out_dir):
+ path = os.path.relpath(path, out_dir)
+ depfile_content[output].add(path)
+
+ with FileWriter(args.depfile_path) as stream:
+ for output, inputs in sorted(depfile_content.items()):
+ stream.write(f'{output}: {" ".join(sorted(inputs))}\n')
+
+
+def compile_module(args, extras_args, build_signature):
+ """Compiles Swift module according to `args`."""
+ for path in (args.target_out_dir, os.path.dirname(args.header_path)):
+ ensure_directory(path)
+
+ # Write the $module-OutputFileMap.json file.
+ output_file_map = generate_output_file_map(args)
+ output_file_map_path = os.path.join(args.target_out_dir,
+ f'{args.module_name}-OutputFileMap.json')
+
+ with FileWriter(output_file_map_path) as stream:
+ json.dump(output_file_map, stream, indent=' ', sort_keys=True)
+
+ # Invoke Swift compiler.
+ with create_build_cache_dir(args, build_signature) as build_cache_dir:
+ invoke_swift_compiler(args,
+ extras_args,
+ build_cache_dir=build_cache_dir,
+ output_file_map=output_file_map_path)
+
+ # Generate the depfile.
+ generate_depfile(args, output_file_map)
def main(args):
- parser = argparse.ArgumentParser(add_help=False)
- parser.add_argument('-module-name', help='name of the Swift module')
- parser.add_argument('-include',
- '-I',
+ parser = argparse.ArgumentParser(allow_abbrev=False, add_help=False)
+
+ # Required arguments.
+ parser.add_argument('-module-name',
+ required=True,
+ help='name of the Swift module')
+
+ parser.add_argument('-src-dir',
+ required=True,
+ help='path to the source directory')
+
+ parser.add_argument('-target-out-dir',
+ required=True,
+ help='path to the object directory')
+
+ parser.add_argument('-header-path',
+ required=True,
+ help='path to the generated header file')
+
+ parser.add_argument('-bridge-header',
+ required=True,
+ help='path to the Objective-C bridge header file')
+
+ parser.add_argument('-depfile-path',
+ required=True,
+ help='path to the output dependency file')
+
+ # Optional arguments.
+ parser.add_argument('-derived-data-dir',
+ help='path to the derived data directory')
+
+ # Required arguments (forwarded to the Swift compiler).
+ parser.add_argument('-target',
+ required=True,
+ dest='target_triple',
+ help='generate code for the given target')
+
+ parser.add_argument('-sdk',
+ required=True,
+ dest='sdk_path',
+ help='path to the iOS SDK')
+
+ parser.add_argument('-const-gather-protocols-file',
+ required=True,
+ help='path to file containing const values protocols')
+
+ # Optional arguments.
+ parser.add_argument('-fix-module-imports',
+ default=False,
+ action='store_true',
+ help='fix module imports in generated header')
+
+ parser.add_argument('-swift-toolchain-path',
+ default='',
+ help='path to the Swift toolchain to use')
+
+ # Optional arguments (forwarded to the Swift compiler).
+ parser.add_argument('-I',
action='append',
dest='include_dirs',
help='add directory to header search path')
+
parser.add_argument('-isystem',
action='append',
dest='system_include_dirs',
help='add directory to system header search path')
- parser.add_argument('sources', nargs='+', help='Swift source file to compile')
- parser.add_argument('-whole-module-optimization',
- action='store_true',
- help='enable whole module optimization')
- parser.add_argument('-object-dir',
- help='path to the generated object files directory')
- parser.add_argument('-pch-output-dir',
- help='path to directory where .pch files are saved')
- parser.add_argument('-module-path', help='path to the generated module file')
- parser.add_argument('-header-path', help='path to the generated header file')
- parser.add_argument('-bridge-header',
- help='path to the Objective-C bridge header')
- parser.add_argument('-depfile', help='path to the generated depfile')
- parser.add_argument('-swift-version',
- help='version of Swift language to support')
- parser.add_argument('-target',
- action='store',
- help='generate code for the given target <triple>')
- parser.add_argument('-sdk', action='store', help='compile against sdk')
+
parser.add_argument('-F',
+ action='append',
dest='framework_dirs',
- action='append',
- help='add dir to framework search path')
+ help='add directory to framework search path')
+
parser.add_argument('-Fsystem',
- '-iframework',
- dest='system_framework_dirs',
action='append',
- help='add dir to system framework search path')
- parser.add_argument('-root-dir',
- dest='root_dir',
- action='store',
- required=True,
- help='path to the root of the repository')
- parser.add_argument('-swift-toolchain-path',
- default='',
- action='store',
- dest='swift_toolchain_path',
- help='path to the root of the Swift toolchain')
+ dest='system_framework_dirs',
+ help='add directory to system framework search path')
+
+ parser.add_argument('-D',
+ action='append',
+ dest='defines',
+ help='add preprocessor define')
+
+ parser.add_argument('-swift-version',
+ default='5',
+ help='version of the Swift language')
+
+ parser.add_argument('-whole-module-optimization',
+ default=False,
+ action='store_true',
+ help='enable whole module optimisation')
+
parser.add_argument('-file-compilation-dir',
- default='',
- action='store',
- help='compilation directory to embed in the debug info')
- parser.add_argument('-enable-cxx-interop',
- dest='enable_cxx_interop',
- action='store_true',
- help='allow importing C++ modules into Swift')
- parser.add_argument('-fix-module-imports',
- action='store_true',
- help='enable hack to fix module imports')
- parser.add_argument('-swiftc-version',
- default='',
- action='store',
- help='version of swiftc compiler')
- parser.add_argument('-xcode-version',
- default='',
- action='store',
- help='version of xcode')
+ help='compilation directory to embed in debug info')
+
+ # Positional arguments.
+ parser.add_argument('sources',
+ nargs='+',
+ help='Swift source files to compile')
parsed, extras = parser.parse_known_args(args)
- with tempfile.TemporaryDirectory() as tmpdir:
- compile_module(parsed.module_name, parsed.sources, parsed, extras, tmpdir)
+ compile_module(parsed, extras, build_signature(os.environ, args))
if __name__ == '__main__':
