public/nixpkgs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
284,193 Commits 1 Branch 0 Tags
5259a2e5da65608e246af51492e00bc02ca9855c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Graham Christensen 5259a2e5da Merge pull request #119996 from DeterminateSystems/sasquatch 
sasquatch: 4.3 -> 4.4
 2021-04-23 11:41:23 -04:00
Graham Christensen 3ac9de55b0 Merge pull request #120304 from DeterminateSystems/yargen 
yarGen: patch get_abs_path function, install strings.xml
 2021-04-23 10:29:37 -04:00
Graham Christensen c86bd27503 Merge pull request #119664 from eyJhb/displaylink-5.4 
displaylink: 5.3.1 -> 5.4.0 (+evdi)
 2021-04-22 13:36:31 +00:00
Graham Christensen a59644aeb6 Merge pull request #119999 from DeterminateSystems/honeytrap 
honeytrap: init at unstable-2020-12-10
 2021-04-22 00:15:07 +00:00
Graham Christensen 20313e8b73 Merge pull request #119906 from DeterminateSystems/team/determinate.systems 
Teams: create a Determinate Systems team
 2021-04-19 18:38:08 -04:00
Graham Christensen 777286310b teams: add cole-h to the determinatesystems team 2021-04-19 18:03:11 -04:00
Graham Christensen 705227c558 teams: init determinatesystems 2021-04-19 18:03:08 -04:00
Graham Christensen ffd29c76f6 Merge pull request #117928 from ymatsiuk/fprintd-tod 
nixos/fprintd: add TOD support
 2021-04-15 11:51:09 -04:00
Graham Christensen a009dfc1af Merge pull request #118400 from mweinelt/revert-zigbee2mqtt 
Revert "zigbee2mqtt: 1.16.2 -> 1.18.2"
 2021-04-03 13:18:58 +00:00
Graham Christensen 83fccd0edb Merge pull request #104915 from grahamc/fprintd-libfprint-tod 
{fprintd,libfprint}-tod: init, plus  libfprint-2-tod1-goodix: init at 0.0.6
 2021-03-30 14:55:24 +00:00
Graham Christensen 65674c24f8 libfprint-2-tod1-goodix: init at 0.0.6 2021-03-30 14:36:25 +00:00
Graham Christensen b0a4d392b9 fprintd-tod: init from fprintd 2021-03-30 14:36:24 +00:00
Graham Christensen 76558714f3 libfprint-tod: init at 1.90.7+tod1 2021-03-30 14:36:19 +00:00
Graham Christensen 60dd94fb7e Merge pull request #117588 from mweinelt/openssl 
openssl: 1.1.1j -> 1.1.1k
 2021-03-25 10:16:50 -04:00
Graham Christensen 7eda163eac Merge pull request #115310 from mweinelt/grub2 
grub: 2.0.4 -> 2.0.6-rc1
 2021-03-22 22:34:14 -04:00
Graham Christensen 7b69e62cb0 Merge pull request #116000 from sternenseemann/mcfgthreads-fix-eval 
mcfgthreads: fix syntax
 2021-03-11 18:00:20 -05:00
Graham Christensen 94a291a5b7 multipath-tools: fix the service path to multipathd 2021-03-11 10:31:42 -05:00
Graham Christensen 0b52c40240 nixos/systemd: bring in modprobe@.service 
It is broken: it looks for /sbin/modprobe, but it works anyway??
 2021-03-09 12:38:54 -05:00
Graham Christensen e7d9750b5c systemd: correct path to modprobe@.service 
The unit hard-codes a path to modprobe which is obviously invalid.
 2021-03-09 12:38:54 -05:00
Graham Christensen c5ae852c37 Merge pull request #114874 from cole-h/ssm-agent 
ssm-agent: 2.3.1319.0 -> 3.0.755.0
 2021-03-06 20:22:48 -05:00
Graham Christensen 1b1bdeac67 Merge pull request #115024 from cole-h/hydra-dbi 
nixos/hydra: append `application_name` to HYDRA_DBI
 2021-03-04 10:06:43 -05:00
Graham ChristensenandCole Helbling 820cc72b52 ssm-agent: 2.3.1319.0 -> 3.0.755.0 
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
 2021-03-02 10:51:12 -08:00
Graham Christensen 9a7fda31bb Merge pull request #113365 from rmcgibbo/ena 
kernelPackages.ena: 2.3.0 -> 2.4.1
 2021-02-23 11:08:34 -05:00
Graham Christensen 3ef281c3a2 Merge pull request #113176 from danieldk/cargoDepsName 
buildRustPackage: add cargoDepsName attribute
 2021-02-23 10:02:45 -05:00
Graham Christensen 744de41d10 Merge pull request #113702 from grahamc/perlpkgs 
perlPackages.{gotofile,LongJump,Test2PluginMemUsage,Test2PluginUUID,Test2Harness}: init
 2021-02-19 16:54:55 -05:00
Graham Christensen 6a53819136 perlPackages.Test2Harness: init at 1.000042 2021-02-19 11:23:45 -05:00
Graham Christensen 5e6c9ffac8 Merge pull request #113687 from grahamc/perlpkgs 
perlPackages.{FunctionParameters,TieHashMethod,TestPostgreSQL}: init
 2021-02-19 11:14:26 -05:00
Graham Christensen 5a6c62b211 perlPackages.Test2PluginUUID: init at 0.002001 2021-02-19 11:13:54 -05:00
Graham Christensen 974155138c perlPackages.Test2PluginMemUsage: init at 0.002003 2021-02-19 11:13:40 -05:00
Graham Christensen 55b70353ce perlPackages.LongJump: init at 0.000001 2021-02-19 11:12:42 -05:00
Graham Christensen 29fe085490 perlPackages.gotofile: init at 0.005 2021-02-19 11:12:18 -05:00
Graham Christensen 4f07645f6f perlPackages.TestPostgreSQL: init at 1.27 2021-02-19 09:30:20 -05:00
Graham Christensen 2058e64c9c perlPackages.TieHashMethod: init at 0.02 2021-02-19 09:30:20 -05:00
Graham Christensen d9736c984c perlPackages.FunctionParameters: init at 2.001003 2021-02-19 09:30:17 -05:00
Graham Christensen 86d5d9650d Merge pull request #111413 from dotlambda/vorta-0.7.2 
vorta: 0.7.1 -> 0.7.2
 2021-01-31 20:56:27 -05:00
Graham Christensen b1d651e989 Merge pull request #111494 from rien/master 
rage: skip check if not x86
 2021-01-31 20:52:14 -05:00
Graham Christensen 5b896d3797 Merge pull request #111513 from devzero/master 
signal-cli: 0.6.12 -> 0.7.4
 2021-01-31 20:38:57 -05:00
Graham Christensen 7092dd52f8 amazonImage: Upload disks as GP3 for cheaper & faster IO (#109027) 
GP3 is always faster and cheaper than GP2, so sticking to GP2 is
leaving money on the table.

https://cloudwiry.com/ebs-gp3-vs-gp2-pricing-comparison/
 2021-01-11 13:54:40 -05:00
Graham Christensen b63aa4ccb2 Deshaw: create team and reassign limeytexan's maintainership (#108678) 
* team: add deshaw

* maintainership: reassign limeytexan's packages to the deshaw team
 2021-01-07 10:44:08 -05:00
Graham Christensen 2bb0558d0a aws-nuke: init at 2.14.0 2021-01-04 14:43:00 -05:00
Graham Christensen 09b571cfe7 Merge pull request #107443 from nima2007/zchunk_work 
zchunk: 1.1.6-> 1.1.8 & Add darwin support
 2020-12-29 14:31:39 -05:00
Graham Christensen 14c2379fa7 Merge pull request #107052 from drewrisinger/dr-pr-vscode-arm 
vscode, vscodium: 1.51.1 -> 1.52.1 & add aarch64-linux, armv7l-linux builds
 2020-12-28 16:55:48 -05:00
Graham Christensen 9612cb9092 Merge pull request #84062 from abathur/fix/yadm_src_usage 
yadm: install from build directory instead of $src
 2020-12-15 10:39:20 -05:00
Graham Christensen 98830e4438 Merge pull request #105790 from grahamc/tf-packet 
terraform-providers.packet: v2.9.0 -> v3.2.0
 2020-12-03 08:28:53 -05:00
Graham Christensen 48413ef09d terraform-providers.packet: v2.9.0 -> v3.2.0 2020-12-03 08:10:56 -05:00
Graham Christensen f18e57f3bf Merge pull request #105452 from zimbatm/no-extend 
tree-wide: do not use pkgs.extend in nixpkgs
 2020-11-30 13:55:33 -05:00
Graham Christensen d9c3f13df3 Merge pull request #104776 from grahamc/utillinux 
utillinux: rename to util-linux
 2020-11-24 15:14:36 -05:00
Graham Christensen bc49a0815a utillinux: rename to util-linux 2020-11-24 12:42:06 -05:00
Graham Christensen 22965d84d5 sssd: fix build on glibc-2.32 
For https://hydra.nixos.org/build/128488320 which errored with:

    src/responder/nss/nss_cmd.c:733:16: error: 'nss_setnetgrent' redeclared as different kind of symbol
      733 | static errno_t nss_setnetgrent(struct cli_ctx *cli_ctx,
          |                ^~~~~~~~~~~~~~~
    In file included from ./src/sss_client/sss_cli.h:28,
                     from ./src/db/sysdb.h:27,
                     from src/responder/nss/nss_cmd.c:26:
    /nix/store/a2n8nrsf215x01a7fv8l94crdjwf69pa-glibc-2.32-dev/include/nss.h:184:25: note: previous declaration of 'nss_setnetgrent' was here
      184 | typedef enum nss_status nss_setnetgrent (const char *, struct __netgrent *);
          |                         ^~~~~~~~~~~~~~~
    make[2]: *** [Makefile:17818: src/responder/nss/nss_cmd.o] Error 1
    make[2]: Leaving directory '/build/sssd-1.16.4'
    make[1]: *** [Makefile:33824: all-recursive] Error 1
    make[1]: Leaving directory '/build/sssd-1.16.4'
    make: *** [Makefile:10123: all] Error 2
 2020-11-24 09:50:33 -05:00
Graham Christensen b8a78e5b2a Merge pull request #104702 from NixOS/revert-104484-danderson/influxdb2 
Revert "influxdb2: init at v2.0.2"
 2020-11-23 14:56:14 -05:00
Graham Christensen fbc4af5725 Revert "influxdb2: init at v2.0.2 (#104484)" 
This reverts commit ecbe92e576.
 2020-11-23 14:55:26 -05:00
Graham Christensen 1ee1134cb1 Merge pull request #104456 from endgame/refresh-instance-metadata-on-boot 
Refresh instance metadata on boot
 2020-11-22 08:23:14 -05:00
Graham Christensen 75d7828724 Merge pull request #98544 from Mic92/unfuck-update-user-group 
nixos/update-user-groups: Fix encoding issues + atomic writes
 2020-11-20 10:28:52 -05:00
Graham Christensen e52ecd43c5 Merge pull request #104335 from grahamc/amazon-image-fixup2 
nixos: release-combined: only build the amazon image for x86_64,aarch…
 2020-11-19 21:09:11 -05:00
Graham Christensen b339462460 nixos: release-combined: only build the amazon image for x86_64,aarch64-linux -- exclude i686 2020-11-19 19:34:23 -05:00
Graham Christensen 1115df837e Merge pull request #104322 from grahamc/amazon-image 
nixos/release-small: add amazonImage
 2020-11-19 18:45:07 -05:00
Graham Christensen 1ef139f3b0 nixos/release-small: add amazonImage 
fixup breakage from #104193
 2020-11-19 17:45:40 -05:00
Graham Christensen 7fa7bf2fda Merge pull request #104193 from grahamc/ec2-metadata-imdsv2 
NixOS EC2 AMI: Support IMDSv2
 2020-11-19 16:11:32 -05:00
Graham Christensen 0d87ce610e nixos: release: add amazonImage as a channel blocker 2020-11-19 13:56:55 -05:00
Graham Christensen f2cfecdec3 nixos ami: preflight the imds token 
According to Freenode's ##AWS, the metadata server can sometimes
take a few moments to get its shoes on, and the very first boot
of a machine can see failed requests for a few moments.
 2020-11-19 13:56:44 -05:00
Graham Christensen 83ea88e03f nixos: ec2 ami: support IMDSv2 
AWS's metadata service has two versions. Version 1 allowed plain HTTP
requests to get metadata. However, this was frequently abused when a
user could trick an AWS-hosted server in to proxying requests to the
metadata service. Since the metadata service is frequently used to
generate AWS access keys, this is pretty gnarly. Version two is
identical except it requires the caller to request a token and provide
it on each request.

Today, starting a NixOS AMI in EC2 where the metadata service is
configured to only allow v2 requests fails: the user's SSH key is not
placed, and configuration provided by the user-data is not applied.
The server is useless. This patch addresses that.

Note the dependency on curl is not a joyful one, and it expand the
initrd by 30M. However, see the added comment for more information
about why this is needed. Note the idea of using `echo` and `nc` are
laughable. Don't do that.
 2020-11-19 13:00:56 -05:00
Graham Christensen 21339b41bf nixos: openstack: have its own metadata fetcher expression 
These two APIs have diverged over time and are no longer compatible.
 2020-11-18 11:42:32 -05:00
Graham Christensen f9121be82e Merge pull request #103307 from grahamc/zfs-scripts 
zfs: extend zpool scripts with :$PATH, remove sudo
 2020-11-10 10:20:02 -05:00
Graham Christensen b3d4f5dada zfs: extend zpool scripts with :$PATH, remove sudo 
sudo the package is not suid, so remove it from the PATH. Extend
PATH with $PATH to fetch sudo from the outer PATH. Move the generation
of `path` in to Nix, so the sed expression can use single quotes,
which prevents $PATH being interpolated at build-time.
 2020-11-10 08:50:22 -05:00
Graham Christensen 43b99f23b3 Merge pull request #102532 from grahamc/nixUnstable-bump 
nixUnstable: 3.0pre20201020_e0ca98c -> 2.4pre20201102_550e11f
 2020-11-02 17:15:14 -05:00
Graham Christensen f4f9f307a4 nixUnstable: 3.0pre20201020_e0ca98c -> 2.4pre20201102_550e11f 2020-11-02 16:53:06 -05:00
Graham Christensen 75a2bc94fa Merge pull request #101192 from grahamc/nixpkgs-location-basic-auth 
nginx: support basic auth in location blocks
 2020-11-02 09:44:54 -05:00
Graham Christensen 3361a037b9 nginx: add a warning that nginx's basic auth isn't very good. 2020-11-02 08:16:01 -05:00
Graham Christensen a4b86b2bf5 nginx: test basic auth 2020-11-02 08:16:01 -05:00
Graham Christensen c7bf3828f0 nginx: add basic auth support for locations 2020-11-02 08:16:00 -05:00
Graham Christensen 33cf4f0e8e nginx: factor out the generation of basic auth generation 2020-11-02 08:16:00 -05:00
Graham Christensen 800ebd4561 Merge pull request #102201 from danderson/danderson/add-cli-to-path 
nixos/tailscale: add tailscale to environment.systemPackages.
 2020-10-30 21:22:16 -04:00
Graham Christensen 38a394bdee Merge pull request #102174 from grahamc/ami-root-use-gpt 
AMI root partition table: use GPT to support >2T partitions
 2020-10-30 16:14:37 -04:00
Graham Christensen 860a3a23c6 Merge pull request #102175 from grahamc/ami-random 
amazon-image: random.trust_cpu=on to cut 10s from boot
 2020-10-30 16:13:41 -04:00
Graham Christensen c06b97175b Merge pull request #102173 from grahamc/create-amis 
create-amis.sh: fixup shellcheck issues, improve error logging, and add configurable service names
 2020-10-30 16:13:18 -04:00
Graham Christensen 82578fc725 Merge pull request #102172 from grahamc/stage-1-datestamps 
stage-1: add datestamps to logs
 2020-10-30 16:13:02 -04:00
Graham Christensen b34cf366aa Merge pull request #102171 from grahamc/faster-ext-resize 
stage-1: modprobe ext{2,3,4} before resizing (so resizing takes less than 45 minutes)
 2020-10-30 16:12:50 -04:00
Graham Christensen d77ddf2a40 nixos.amazonAmi: use legacy+gpt disk images to support partitions >2T 2020-10-30 15:50:25 -04:00
Graham Christensen d78aa080f5 make-disk-image: support legacy+gpt 2020-10-30 15:50:24 -04:00
Graham Christensen c851030763 amazon-image: random.trust_cpu=on to cut 10s from boot 
Ubuntu and other distros already have this set via kernel config.
 2020-10-30 13:45:19 -04:00
Graham ChristensenandCole Helbling 74a577b293 create-amis: improve wording around the service name's IAM role 
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
 2020-10-30 12:40:17 -04:00
Graham Christensen ece5c0f304 stage-1: modprobe ext{2,3,4} before resizing 
I noticed booting a system with an ext4 root which expanded to 5T took
quite a long time (12 minutes in some cases, 43(!) in others.)

I changed stage-1 to run `resize2fs -d 62` for extra debug output and
timing information. It revealed the adjust_superblock step taking
almost all of the time:

    [Fri Oct 30 11:10:15 UTC 2020] zero_high_bits_in_metadata: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
    [Fri Oct 30 11:21:09 UTC 2020] adjust_superblock: Memory used: 396k/4556k (295k/102k), time: 654.21/ 0.59/ 5.13

but when I ran resize2fs on a disk with the identical content growing
to the identical target size, it would only take about 30 seconds. I
looked at what happened between those two steps in the fast case with
strace and found:

```
   235	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=1795}, ru_stime={tv_sec=0, tv_usec=3590}, ...}) = 0
   236	write(1, "zero_high_bits_in_metadata: Memo"..., 84zero_high_bits_in_metadata: Memory used: 132k/0k (72k/61k), time:  0.00/ 0.00/ 0.00
   237	) = 84
   238	gettimeofday({tv_sec=1604061278, tv_usec=480147}, NULL) = 0
   239	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=1802}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   240	gettimeofday({tv_sec=1604061278, tv_usec=480192}, NULL) = 0
   241	mmap(NULL, 2564096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa3c7355000
   242	access("/sys/fs/ext4/features/lazy_itable_init", F_OK) = 0
   243	brk(0xf85000)                           = 0xf85000
   244	brk(0xfa6000)                           = 0xfa6000
   245	gettimeofday({tv_sec=1604061278, tv_usec=538828}, NULL) = 0
   246	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=58720}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   247	write(1, "adjust_superblock: Memory used: "..., 79adjust_superblock: Memory used: 396k/2504k (305k/92k), time:  0.06/ 0.06/ 0.00
   248	) = 79
   249	gettimeofday({tv_sec=1604061278, tv_usec=539119}, NULL) = 0
   250	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=58812}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   251	gettimeofday({tv_sec=1604061279, tv_usec=939}, NULL) = 0
   252	getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=520411}, ru_stime={tv_sec=0, tv_usec=3603}, ...}) = 0
   253	write(1, "fix_uninit_block_bitmaps 2: Memo"..., 88fix_uninit_block_bitmaps 2: Memory used: 396k/2504k (305k/92k), time:  0.46/ 0.46/ 0.00
   254	) = 88
```

In particular the access to /sys/fs seemed interesting. Looking
at the source of resize2fs:

```
[root@ip-172-31-22-182:~/e2fsprogs-1.45.5]# rg -B2 -A1 /sys/fs/ext4/features/lazy_itable_init .
./resize/resize2fs.c
923-	if (getenv("RESIZE2FS_FORCE_LAZY_ITABLE_INIT") ||
924-	    (!getenv("RESIZE2FS_FORCE_ITABLE_INIT") &&
925:	     access("/sys/fs/ext4/features/lazy_itable_init", F_OK) == 0))
926-		lazy_itable_init = 1;
```

I confirmed /sys is mounted, and then found a bug suggesting the
ext4 module is maybe not loaded:
https://bugzilla.redhat.com/show_bug.cgi?id=1071909

My home server doesn't have ext4 loaded and had 3T to play with, so
I tried (and succeeded with) replicating the issue locally:

```
[root@kif:/scratch]# lsmod | grep -i ext

[root@kif:/scratch]# zfs create -V 3G rpool/scratch/ext4

[root@kif:/scratch]# time mkfs.ext4 /dev/zvol/rpool/scratch/ext4
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 560a4a8f-93dc-40cc-97a5-f10049bf801f
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

real	0m2.261s
user	0m0.000s
sys	0m0.025s

[root@kif:/scratch]# zfs set volsize=3T rpool/scratch/ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4
resize2fs 1.45.5 (07-Jan-2020)
fs has 11 inodes, 1 groups required.
fs requires 16390 data blocks.
With 1 group(s), we have 22234 blocks available.
Last group's overhead is 10534
Need 16390 data blocks in last group
Final size of last group is 26924
Estimated blocks needed: 26924
Extents safety margin: 49
Resizing the filesystem on /dev/zvol/rpool/scratch/ext4 to 805306368 (4k) blocks.
read_bitmaps: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
read_bitmaps: I/O read: 1MB, write: 0MB, rate: 3802.28MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
resize_group_descriptors: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
move_bg_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
zero_high_bits_in_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
```

here it got stuck for quite some time ... straceing this 20 minutes in revealed this in a tight loop:

```
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
fallocate(3, FALLOC_FL_ZERO_RANGE, 2222649901056, 2097152) = 0
fsync(3)                                = 0
```

it finally ended 43(!) minutes later:

```
adjust_superblock: Memory used: 264k/3592k (210k/55k), time: 2554.03/ 0.16/15.07
fix_uninit_block_bitmaps 2: Memory used: 264k/3592k (210k/55k), time:  0.16/ 0.16/ 0.00
blocks_to_move: Memory used: 264k/3592k (211k/54k), time:  0.00/ 0.00/ 0.00
Number of free blocks: 755396/780023556, Needed: 0
block_mover: Memory used: 264k/3592k (216k/49k), time:  0.05/ 0.01/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 18.68MB/s
inode_scan_and_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
move_itables: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
calculate_summary_stats: Memory used: 264k/3592k (216k/49k), time: 16.35/16.35/ 0.00
fix_resize_inode: Memory used: 264k/3592k (222k/43k), time:  0.04/ 0.00/ 0.00
fix_resize_inode: I/O read: 1MB, write: 0MB, rate: 22.80MB/s
fix_sb_journal_backup: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 264k/3592k (222k/43k), time: 2570.90/16.68/15.07
overall resize2fs: I/O read: 1MB, write: 1MB, rate: 0.00MB/s
The filesystem on /dev/zvol/rpool/scratch/ext4 is now 805306368 (4k) blocks long.

real	43m1.943s
user	0m16.761s
sys	0m15.069s
```

I then cleaned up and recreated the zvol, loaded the ext4 module, created the ext4 fs,
resized the volume, and resize2fs'd and it went quite quickly:

```
[root@kif:/scratch]# zfs destroy rpool/scratch/ext4

[root@kif:/scratch]# zfs create -V 3G rpool/scratch/ext4

[root@kif:/scratch]# modprobe ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4

[root@kif:/scratch]# time mkfs.ext4 /dev/zvol/rpool/scratch/ext4
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 786432 4k blocks and 196608 inodes
Filesystem UUID: 5b415f2f-a8c4-4ba0-ac1d-78860de77610
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

real	0m1.013s
user	0m0.001s
sys	0m0.023s

[root@kif:/scratch]# zfs set volsize=3T rpool/scratch/ext4

[root@kif:/scratch]# time resize2fs -d 62 /dev/zvol/rpool/scratch/ext4
resize2fs 1.45.5 (07-Jan-2020)
fs has 11 inodes, 1 groups required.
fs requires 16390 data blocks.
With 1 group(s), we have 22234 blocks available.
Last group's overhead is 10534
Need 16390 data blocks in last group
Final size of last group is 26924
Estimated blocks needed: 26924
Extents safety margin: 49
Resizing the filesystem on /dev/zvol/rpool/scratch/ext4 to 805306368 (4k) blocks.
read_bitmaps: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
read_bitmaps: I/O read: 1MB, write: 0MB, rate: 3389.83MB/s
fix_uninit_block_bitmaps 1: Memory used: 132k/0k (63k/70k), time:  0.00/ 0.00/ 0.00
resize_group_descriptors: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
move_bg_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
zero_high_bits_in_metadata: Memory used: 132k/0k (68k/65k), time:  0.00/ 0.00/ 0.00
adjust_superblock: Memory used: 264k/1540k (210k/55k), time:  0.02/ 0.02/ 0.00
fix_uninit_block_bitmaps 2: Memory used: 264k/1540k (210k/55k), time:  0.15/ 0.15/ 0.00
blocks_to_move: Memory used: 264k/1540k (211k/54k), time:  0.00/ 0.00/ 0.00
Number of free blocks: 755396/780023556, Needed: 0
block_mover: Memory used: 264k/3592k (216k/49k), time:  0.01/ 0.01/ 0.00
block_mover: I/O read: 1MB, write: 0MB, rate: 157.11MB/s
inode_scan_and_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
inode_ref_fix: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00
move_itables: Memory used: 264k/3592k (216k/49k), time:  0.00/ 0.00/ 0.00

calculate_summary_stats: Memory used: 264k/3592k (216k/49k), time: 16.20/16.20/ 0.00
fix_resize_inode: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
fix_resize_inode: I/O read: 1MB, write: 0MB, rate: 5319.15MB/s
fix_sb_journal_backup: Memory used: 264k/3592k (222k/43k), time:  0.00/ 0.00/ 0.00
overall resize2fs: Memory used: 264k/3592k (222k/43k), time: 16.45/16.38/ 0.00
overall resize2fs: I/O read: 1MB, write: 1MB, rate: 0.06MB/s
The filesystem on /dev/zvol/rpool/scratch/ext4 is now 805306368 (4k) blocks long.

real	0m17.908s
user	0m16.386s
sys	0m0.079s
```

Success!
 2020-10-30 12:18:23 -04:00
Graham Christensen a179781696 stage-1: add datestamps to logs 
When the stage-1 logs get imported in to the journal, they all get
loaded with the same timestamp. This makes it difficult to identify
what might be taking a long time in early boot.
 2020-10-30 12:16:35 -04:00
Graham Christensen 2bf1fc0345 create-amis: allow customizing the service role name 
The complete setup on the AWS end can be configured
with the following Terraform configuration. It generates
a ./credentials.sh which I just copy/pasted in to the
create-amis.sh script near the top. Note: the entire stack
of users and bucket can be destroyed at the end of the
import.

    variable "region" {
      type = string
    }
    variable "availability_zone" {
      type = string
    }

    provider "aws" {
      region = var.region
    }

    resource "aws_s3_bucket" "nixos-amis" {
      bucket_prefix = "nixos-amis-"
      lifecycle_rule {
        enabled = true
        abort_incomplete_multipart_upload_days = 1
        expiration {
          days = 7
        }
      }
    }

    resource "local_file" "credential-file" {
      file_permission = "0700"
      filename = "${path.module}/credentials.sh"
      sensitive_content = <<SCRIPT
    export service_role_name="${aws_iam_role.vmimport.name}"
    export bucket="${aws_s3_bucket.nixos-amis.bucket}"
    export AWS_ACCESS_KEY_ID="${aws_iam_access_key.uploader.id}"
    export AWS_SECRET_ACCESS_KEY="${aws_iam_access_key.uploader.secret}"
    SCRIPT
    }

    # The following resources are for the *uploader*
    resource "aws_iam_user" "uploader" {
      name = "nixos-amis-uploader"
    }

    resource "aws_iam_access_key" "uploader" {
      user = aws_iam_user.uploader.name
    }

    resource "aws_iam_user_policy" "upload-to-nixos-amis" {
      user = aws_iam_user.uploader.name

      policy = data.aws_iam_policy_document.upload-policy-document.json
    }

    data "aws_iam_policy_document" "upload-policy-document" {
      statement {
        effect = "Allow"

        actions = [
          "s3:ListBucket",
          "s3:GetBucketLocation",
        ]

        resources = [
          aws_s3_bucket.nixos-amis.arn
        ]
      }

      statement {
        effect = "Allow"

        actions = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
        ]

        resources = [
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }

      statement {
        effect = "Allow"
        actions = [
          "ec2:ImportSnapshot",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:DescribeImportSnapshotTasks",
          "ec2:RegisterImage",
          "ec2:DescribeImages"
        ]
        resources = [
          "*"
        ]
      }
    }

    # The following resources are for the *vmimport service user*
    # See: https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html#vmimport-role
    resource "aws_iam_role" "vmimport" {
      assume_role_policy = data.aws_iam_policy_document.vmimport-trust.json
    }

    resource "aws_iam_role_policy" "vmimport-access" {
      role = aws_iam_role.vmimport.id
      policy = data.aws_iam_policy_document.vmimport-access.json
    }

    data "aws_iam_policy_document" "vmimport-access" {
      statement {
        effect = "Allow"
        actions = [
          "s3:GetBucketLocation",
          "s3:GetObject",
          "s3:ListBucket",
        ]
        resources = [
          aws_s3_bucket.nixos-amis.arn,
          "${aws_s3_bucket.nixos-amis.arn}/*"
        ]
      }
      statement {
        effect = "Allow"
        actions = [
          "ec2:ModifySnapshotAttribute",
          "ec2:CopySnapshot",
          "ec2:RegisterImage",
          "ec2:Describe*"
        ]
        resources = [
          "*"
        ]
      }
    }

    data "aws_iam_policy_document" "vmimport-trust" {
      statement {
        effect = "Allow"
        principals {
          type = "Service"
          identifiers = [ "vmie.amazonaws.com" ]
        }

        actions = [
          "sts:AssumeRole"
        ]

        condition {
          test = "StringEquals"
          variable = "sts:ExternalId"
          values = [ "vmimport" ]
        }
      }
    }
 2020-10-30 12:12:08 -04:00
Graham Christensen e253de8a77 create-amis.sh: log the full response if describing the import snapshot tasks fails 2020-10-30 12:08:01 -04:00
Graham Christensen f92a883ddb nixos ec2/create-amis.sh: shellcheck: $ is not needed in arithmetic 2020-10-30 12:08:01 -04:00
Graham Christensen 7dac8470cf nixos ec2/create-amis.sh: shellcheck: explicitly make the additions to block_device_mappings single strings 2020-10-30 12:08:00 -04:00
Graham Christensen a66a22ca54 nixos ec2/create-amis.sh: shellcheck: read without -r mangles backslashes 2020-10-30 12:08:00 -04:00
Graham Christensen baf7ed3f24 nixos ec2/create-amis.sh: shellcheck: SC2155: Declare and assign separately to avoid masking return values. 2020-10-30 12:07:59 -04:00
Graham Christensen f5994c208d nixos ec2/create-amis.sh: shellcheck: quote state_dir reference 2020-10-30 12:07:59 -04:00
Graham Christensen c76692192a nixos ec2/create-amis.sh: shellcheck: quote region references 2020-10-30 12:07:49 -04:00
Graham Christensen 4770d412c9 Merge pull request #101470 from endocrimes/dani/bump-p4 
p4: 2020.1.1991450 -> 2020.1.2007551
 2020-10-23 12:17:10 -04:00
Graham Christensen ad9c8e6f04 Merge pull request #101416 from endocrimes/dani/tailscale-version 
tailscale: Include version in build flags
 2020-10-23 09:47:28 -04:00
Graham Christensen 8726057fbd Merge pull request #84617 from xfix/remove-rasm-march-native 
rasm: remove -march=native from compilation flags
 2020-10-14 12:02:36 -04:00
Graham Christensen 961a99db3c oil: 0.8.pre6 -> 0.8.1 2020-10-02 08:42:23 -04:00
Graham Christensen d9a93852d4 nixos-rebuild: support --upgrade-all and document --upgrade (#83327) 2020-09-25 17:22:11 +02:00
Graham Christensen fe3a49fd90 Merge pull request #98272 from tnias/nixos_doc_dev_test_link 
nixos/doc: fix dead link
 2020-09-19 09:21:12 -04:00
Graham Christensen 6e98b8ff93 Merge pull request #97790 from Infinisil/release-nixpkgsArgs 
release*: Support nixpkgsArgs for all release-*.nix files
 2020-09-16 09:17:02 -04:00
Graham Christensen ff42f461f0 Merge pull request #93161 from cole-h/rm-tmp 
Remove tmp file from tarball job
 2020-07-15 07:31:31 -04:00
Graham Christensen 0c10b75288 Merge pull request #93110 from grahamc/systemd-executable 
stage-2: parameterized systemd executable
 2020-07-14 15:45:00 -04:00