[PATCH] qrap: Pass PCI device numbers to qemu in base 10, not in base 16

Stefano Brivio

21 Feb 2023 21 Feb '23

8:24 p.m.

Alona reports that when libvirt starts qrap (KubeVirt integration) and the domain description leads to more than 10 devices, indices of PCI device descriptors are formatted as hexadecimal, so we end up with things like "pci.a" instead of "pci.10". Reported-by: Alona Paz Fixes: 5307faa05997 ("qrap: Strip network devices from command line, set them up according to machine") Signed-off-by: Stefano Brivio --- qrap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/qrap.c b/qrap.c index 287198e..ff99c89 100644 --- a/qrap.c +++ b/qrap.c @@ -281,11 +281,11 @@ int main(int argc, char **argv) qemu_argv[qemu_argc++] = "-device"; if (!has_json) { snprintf(dev_str, ARG_MAX, - "%s,%s%x%s,netdev=hostnet0,x-txburst=4096", + "%s,%s%i%s,netdev=hostnet0,x-txburst=4096", dev->name, dev->template, i, dev->template_post); } else { snprintf(dev_str, ARG_MAX, - "{\"driver\":\"%s\",%s%x\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", + "{\"driver\":\"%s\",%s%i\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", dev->name, dev->template_json, i, dev->template_json_post); } qemu_argv[qemu_argc++] = dev_str; -- 2.39.1

Show replies by date

David Gibson

22 Feb 22 Feb

1:27 a.m.

On Tue, Feb 21, 2023 at 08:24:25PM +0100, Stefano Brivio wrote:

...

Alona reports that when libvirt starts qrap (KubeVirt integration) and the domain description leads to more than 10 devices, indices of PCI device descriptors are formatted as hexadecimal, so we end up with things like "pci.a" instead of "pci.10".

Reported-by: Alona Paz Fixes: 5307faa05997 ("qrap: Strip network devices from command line, set them up according to machine") Signed-off-by: Stefano Brivio

Reviewed-by: David Gibson

...

--- qrap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/qrap.c b/qrap.c index 287198e..ff99c89 100644 --- a/qrap.c +++ b/qrap.c @@ -281,11 +281,11 @@ int main(int argc, char **argv) qemu_argv[qemu_argc++] = "-device"; if (!has_json) { snprintf(dev_str, ARG_MAX, - "%s,%s%x%s,netdev=hostnet0,x-txburst=4096", + "%s,%s%i%s,netdev=hostnet0,x-txburst=4096", dev->name, dev->template, i, dev->template_post); } else { snprintf(dev_str, ARG_MAX, - "{\"driver\":\"%s\",%s%x\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", + "{\"driver\":\"%s\",%s%i\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", dev->name, dev->template_json, i, dev->template_json_post); } qemu_argv[qemu_argc++] = dev_str;

-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson

Andrea Bolognani

11:40 a.m.

On Tue, Feb 21, 2023 at 08:24:25PM +0100, Stefano Brivio wrote:

...

if (!has_json) { snprintf(dev_str, ARG_MAX, - "%s,%s%x%s,netdev=hostnet0,x-txburst=4096", + "%s,%s%i%s,netdev=hostnet0,x-txburst=4096", dev->name, dev->template, i, dev->template_post); } else { snprintf(dev_str, ARG_MAX, - "{\"driver\":\"%s\",%s%x\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", + "{\"driver\":\"%s\",%s%i\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", dev->name, dev->template_json, i, dev->template_json_post); }

I don't think this is going to work. The problem is that, while PCI buses are indeed named with increasing numbers in integer format (pci.9, pci.10 and so on), PCI slots are addressed using hexadecimal format (0x9, 0xa and so on). libvirt uses this naming convention because it matches QEMU's. So if you have an i440fx machine with many devices, they will all[1] be put on pci.0, the corresponding arguments will look like -device virtio-net-pci,bus=pci.0,addr=0xa and the original version of the code will work correctly. If the machine is q35, however, devices will not be put on pcie.0 directly but rather attached to pcie-root-ports, each one of which creates a separate PCI bus. So the arguments will look like -device virtio-net-pci,bus=pci.10,addr=0x0 where it is the new version of the code that would do the right thing. What I think needs to happen, is that each pci_dev should contain a base value (16 for i440fx and 10 for q35), which is used both in the strtol() call used to parse the command line produced by libvirt and to decide whether %x or %i should be used with snprintf() to generate qrap's own arguments. [1] Up until the point where are too many devices to fit into pci.0 and adding a pci-bridge becomes necessary. I think qrap wouldn't be able to handle this scenario correctly anyway. -- Andrea Bolognani / Red Hat / Virtualization

Stefano Brivio

11:45 a.m.

On Wed, 22 Feb 2023 02:40:32 -0800 Andrea Bolognani wrote:

...

On Tue, Feb 21, 2023 at 08:24:25PM +0100, Stefano Brivio wrote:

...
if (!has_json) { snprintf(dev_str, ARG_MAX, - "%s,%s%x%s,netdev=hostnet0,x-txburst=4096", + "%s,%s%i%s,netdev=hostnet0,x-txburst=4096", dev->name, dev->template, i, dev->template_post); } else { snprintf(dev_str, ARG_MAX, - "{\"driver\":\"%s\",%s%x\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", + "{\"driver\":\"%s\",%s%i\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", dev->name, dev->template_json, i, dev->template_json_post); }

I don't think this is going to work.

The problem is that, while PCI buses are indeed named with increasing numbers in integer format (pci.9, pci.10 and so on), PCI slots are addressed using hexadecimal format (0x9, 0xa and so on). libvirt uses this naming convention because it matches QEMU's.

So if you have an i440fx machine with many devices, they will all[1] be put on pci.0, the corresponding arguments will look like

-device virtio-net-pci,bus=pci.0,addr=0xa

and the original version of the code will work correctly.

If the machine is q35, however, devices will not be put on pcie.0 directly but rather attached to pcie-root-ports, each one of which creates a separate PCI bus. So the arguments will look like

-device virtio-net-pci,bus=pci.10,addr=0x0

where it is the new version of the code that would do the right thing.

Oh, oops, I didn't know that.

...

What I think needs to happen, is that each pci_dev should contain a base value (16 for i440fx and 10 for q35), which is used both in the strtol() call used to parse the command line produced by libvirt and to decide whether %x or %i should be used with snprintf() to generate qrap's own arguments.

Right, I think it makes sense. Will you prepare a patch in this sense or should I try? -- Stefano

Andrea Bolognani

1:47 p.m.

On Wed, Feb 22, 2023 at 11:45:08AM +0100, Stefano Brivio wrote:

...

On Wed, 22 Feb 2023 02:40:32 -0800 Andrea Bolognani wrote:

...
What I think needs to happen, is that each pci_dev should contain a base value (16 for i440fx and 10 for q35), which is used both in the strtol() call used to parse the command line produced by libvirt and to decide whether %x or %i should be used with snprintf() to generate qrap's own arguments.

Right, I think it makes sense. Will you prepare a patch in this sense or should I try?

Sure, I can give it a shot. It might take a bit, because the code is kinda tricky and there is no test suite (that I'm aware of?) for this part. -- Andrea Bolognani / Red Hat / Virtualization

Stefano Brivio

2:02 p.m.

On Wed, 22 Feb 2023 04:47:32 -0800 Andrea Bolognani wrote:

...

On Wed, Feb 22, 2023 at 11:45:08AM +0100, Stefano Brivio wrote:

...
On Wed, 22 Feb 2023 02:40:32 -0800 Andrea Bolognani wrote:

...
What I think needs to happen, is that each pci_dev should contain a base value (16 for i440fx and 10 for q35), which is used both in the strtol() call used to parse the command line produced by libvirt and to decide whether %x or %i should be used with snprintf() to generate qrap's own arguments.

Right, I think it makes sense. Will you prepare a patch in this sense or should I try?

Sure, I can give it a shot. It might take a bit, because the code is kinda tricky and there is no test suite (that I'm aware of?) for this part.

Thanks! Right, sorry, no tests covering qrap at the moment, and the code is horrible. The KubeVirt integration might help if you have an environment ready. -- Stefano

David Gibson

11:27 p.m.

On Wed, Feb 22, 2023 at 02:40:32AM -0800, Andrea Bolognani wrote:

...

On Tue, Feb 21, 2023 at 08:24:25PM +0100, Stefano Brivio wrote:

...
if (!has_json) { snprintf(dev_str, ARG_MAX, - "%s,%s%x%s,netdev=hostnet0,x-txburst=4096", + "%s,%s%i%s,netdev=hostnet0,x-txburst=4096", dev->name, dev->template, i, dev->template_post); } else { snprintf(dev_str, ARG_MAX, - "{\"driver\":\"%s\",%s%x\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", + "{\"driver\":\"%s\",%s%i\"%s,\"netdev\":\"hostnet0\",\"x-txburst\":4096}", dev->name, dev->template_json, i, dev->template_json_post); }

I don't think this is going to work.

The problem is that, while PCI buses are indeed named with increasing numbers in integer format (pci.9, pci.10 and so on), PCI slots are addressed using hexadecimal format (0x9, 0xa and so on). libvirt uses this naming convention because it matches QEMU's.

Actually, I think we're ok. PCI slots are addressed in hex by convention, but AFAICT if you *just* give a slot number, it will accept either decimal or hex (so addr=10 and addr=0xa are equivalent). That's *not* true if you use SS.F format to include the function number - then it expects hex only. But we're not doing that, so so always using decimal should be ok here. Source: set_pci_devfn() in the qemu source Obviously that's a pretty fragile hack, but that's 'qrap' for you. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson

Andrea Bolognani

23 Feb 23 Feb

3:06 p.m.

On Thu, Feb 23, 2023 at 09:27:14AM +1100, David Gibson wrote:

...

On Wed, Feb 22, 2023 at 02:40:32AM -0800, Andrea Bolognani wrote:

...
I don't think this is going to work.

The problem is that, while PCI buses are indeed named with increasing numbers in integer format (pci.9, pci.10 and so on), PCI slots are addressed using hexadecimal format (0x9, 0xa and so on). libvirt uses this naming convention because it matches QEMU's.

Actually, I think we're ok. PCI slots are addressed in hex by convention, but AFAICT if you *just* give a slot number, it will accept either decimal or hex (so addr=10 and addr=0xa are equivalent). That's *not* true if you use SS.F format to include the function number - then it expects hex only. But we're not doing that, so so always using decimal should be ok here.

Source: set_pci_devfn() in the qemu source

Obviously that's a pretty fragile hack, but that's 'qrap' for you.

Yeah, even if that happens to work I'd rather not rely on it, especially since a proper solution doesn't look like it would be a lot of additional effort. I've managed to reproduce the original issue in the context of KubeVirt. I'll hopefully have a patch ready soon. -- Andrea Bolognani / Red Hat / Virtualization

Stefano Brivio

24 Feb 24 Feb

8:14 a.m.

On Thu, 23 Feb 2023 06:06:17 -0800 Andrea Bolognani wrote:

...

On Thu, Feb 23, 2023 at 09:27:14AM +1100, David Gibson wrote:

...
On Wed, Feb 22, 2023 at 02:40:32AM -0800, Andrea Bolognani wrote:

...
I don't think this is going to work.

The problem is that, while PCI buses are indeed named with increasing numbers in integer format (pci.9, pci.10 and so on), PCI slots are addressed using hexadecimal format (0x9, 0xa and so on). libvirt uses this naming convention because it matches QEMU's.

Actually, I think we're ok. PCI slots are addressed in hex by convention, but AFAICT if you *just* give a slot number, it will accept either decimal or hex (so addr=10 and addr=0xa are equivalent). That's *not* true if you use SS.F format to include the function number - then it expects hex only. But we're not doing that, so so always using decimal should be ok here.

Source: set_pci_devfn() in the qemu source

Obviously that's a pretty fragile hack, but that's 'qrap' for you.

Yeah, even if that happens to work I'd rather not rely on it, especially since a proper solution doesn't look like it would be a lot of additional effort.

I've managed to reproduce the original issue in the context of KubeVirt. I'll hopefully have a patch ready soon.

Andrea, allow me to do this: I would push this patch meanwhile, along with the changes for the DNS issue you reported, because that one might impact many users, and I think it makes sense to have a fix out soon. I start thinking it's also part of the issue Paul reported for Podman with pasta here: https://github.com/containers/podman/issues/17074 This patch itself can't hurt, and it changes exactly two letters. As soon as you have something less qrappy we'll go with that (you don't even need to rebase, I'll revert this one on the tree first). -- Stefano

Andrea Bolognani

8:05 p.m.

On Fri, Feb 24, 2023 at 08:14:16AM +0100, Stefano Brivio wrote:

...

On Thu, 23 Feb 2023 06:06:17 -0800 Andrea Bolognani wrote:

...
On Thu, Feb 23, 2023 at 09:27:14AM +1100, David Gibson wrote:

...
On Wed, Feb 22, 2023 at 02:40:32AM -0800, Andrea Bolognani wrote:

...
I don't think this is going to work.

The problem is that, while PCI buses are indeed named with increasing numbers in integer format (pci.9, pci.10 and so on), PCI slots are addressed using hexadecimal format (0x9, 0xa and so on). libvirt uses this naming convention because it matches QEMU's.

Actually, I think we're ok. PCI slots are addressed in hex by convention, but AFAICT if you *just* give a slot number, it will accept either decimal or hex (so addr=10 and addr=0xa are equivalent). That's *not* true if you use SS.F format to include the function number - then it expects hex only. But we're not doing that, so so always using decimal should be ok here.

Source: set_pci_devfn() in the qemu source

Obviously that's a pretty fragile hack, but that's 'qrap' for you.

Yeah, even if that happens to work I'd rather not rely on it, especially since a proper solution doesn't look like it would be a lot of additional effort.

I've managed to reproduce the original issue in the context of KubeVirt. I'll hopefully have a patch ready soon.

Andrea, allow me to do this: I would push this patch meanwhile, along with the changes for the DNS issue you reported, because that one might impact many users, and I think it makes sense to have a fix out soon.

I start thinking it's also part of the issue Paul reported for Podman with pasta here: https://github.com/containers/podman/issues/17074

This patch itself can't hurt, and it changes exactly two letters.

I strongly disagree with this assessment. This patch merely trades one set of issues for another one. In particular, for pc machine types we'd end up producing bus=pci.0,addr=0x10 for slot 10 instead of bus=pci.0,addr=0xa, because the addr=0x part is baked into the template. So the QEMU logic David mentioned above wouldn't kick in at all. More importantly, for q35 machines we'd start producing decimal bus numbers while still parsing the ones present in the original command line as hexadecimal, so things would stop lining up as soon as enough devices are present, meaning that the issue reported by Alona would still exist.

...

As soon as you have something less qrappy we'll go with that (you don't even need to rebase, I'll revert this one on the tree first).

Patches fixing this issue, as well as a few additional ones, are now on the list. I'll follow up on that thread with some considerations related to testing the changes. -- Andrea Bolognani / Red Hat / Virtualization

Stefano Brivio

8:32 p.m.

On Fri, 24 Feb 2023 11:05:00 -0800 Andrea Bolognani wrote:

...

On Fri, Feb 24, 2023 at 08:14:16AM +0100, Stefano Brivio wrote:

...
On Thu, 23 Feb 2023 06:06:17 -0800 Andrea Bolognani wrote:

...
On Thu, Feb 23, 2023 at 09:27:14AM +1100, David Gibson wrote:

...
On Wed, Feb 22, 2023 at 02:40:32AM -0800, Andrea Bolognani wrote:

...
I don't think this is going to work.

The problem is that, while PCI buses are indeed named with increasing numbers in integer format (pci.9, pci.10 and so on), PCI slots are addressed using hexadecimal format (0x9, 0xa and so on). libvirt uses this naming convention because it matches QEMU's.

Actually, I think we're ok. PCI slots are addressed in hex by convention, but AFAICT if you *just* give a slot number, it will accept either decimal or hex (so addr=10 and addr=0xa are equivalent). That's *not* true if you use SS.F format to include the function number - then it expects hex only. But we're not doing that, so so always using decimal should be ok here.

Source: set_pci_devfn() in the qemu source

Obviously that's a pretty fragile hack, but that's 'qrap' for you.

Yeah, even if that happens to work I'd rather not rely on it, especially since a proper solution doesn't look like it would be a lot of additional effort.

I've managed to reproduce the original issue in the context of KubeVirt. I'll hopefully have a patch ready soon.

Andrea, allow me to do this: I would push this patch meanwhile, along with the changes for the DNS issue you reported, because that one might impact many users, and I think it makes sense to have a fix out soon.

I start thinking it's also part of the issue Paul reported for Podman with pasta here: https://github.com/containers/podman/issues/17074

This patch itself can't hurt, and it changes exactly two letters.

I strongly disagree with this assessment. This patch merely trades one set of issues for another one.

In particular, for pc machine types we'd end up producing bus=pci.0,addr=0x10 for slot 10 instead of bus=pci.0,addr=0xa, because the addr=0x part is baked into the template. So the QEMU logic David mentioned above wouldn't kick in at all.

More importantly, for q35 machines we'd start producing decimal bus numbers while still parsing the ones present in the original command line as hexadecimal, so things would stop lining up as soon as enough devices are present, meaning that the issue reported by Alona would still exist.

Oh, okay, sorry, I thought you and David agreed that it actually happens to work. But anyway, nice that it doesn't matter now. :)

...

...
As soon as you have something less qrappy we'll go with that (you don't even need to rebase, I'll revert this one on the tree first).

Patches fixing this issue, as well as a few additional ones, are now on the list. I'll follow up on that thread with some considerations related to testing the changes.

Thanks a lot! I didn't push out commits or a release today because I hit a false positive with cppcheck 2.10 (cppcheck bisected but still trying to grasp the issue), so that will all be for the next week I guess. -- Stefano

David Gibson

25 Feb 25 Feb

2:44 a.m.

On Fri, Feb 24, 2023 at 08:32:56PM +0100, Stefano Brivio wrote: 11;rgb:ffff/ffff/ffff> On Fri, 24 Feb 2023 11:05:00 -0800

...

Andrea Bolognani wrote:

...
On Fri, Feb 24, 2023 at 08:14:16AM +0100, Stefano Brivio wrote:

...
On Thu, 23 Feb 2023 06:06:17 -0800 Andrea Bolognani wrote:

...
On Thu, Feb 23, 2023 at 09:27:14AM +1100, David Gibson wrote:

...
On Wed, Feb 22, 2023 at 02:40:32AM -0800, Andrea Bolognani wrote:

...
I don't think this is going to work.

The problem is that, while PCI buses are indeed named with increasing numbers in integer format (pci.9, pci.10 and so on), PCI slots are addressed using hexadecimal format (0x9, 0xa and so on). libvirt uses this naming convention because it matches QEMU's.

Actually, I think we're ok. PCI slots are addressed in hex by convention, but AFAICT if you *just* give a slot number, it will accept either decimal or hex (so addr=10 and addr=0xa are equivalent). That's *not* true if you use SS.F format to include the function number - then it expects hex only. But we're not doing that, so so always using decimal should be ok here.

Source: set_pci_devfn() in the qemu source

Obviously that's a pretty fragile hack, but that's 'qrap' for you.

Yeah, even if that happens to work I'd rather not rely on it, especially since a proper solution doesn't look like it would be a lot of additional effort.

I've managed to reproduce the original issue in the context of KubeVirt. I'll hopefully have a patch ready soon.

Andrea, allow me to do this: I would push this patch meanwhile, along with the changes for the DNS issue you reported, because that one might impact many users, and I think it makes sense to have a fix out soon.

I start thinking it's also part of the issue Paul reported for Podman with pasta here: https://github.com/containers/podman/issues/17074

This patch itself can't hurt, and it changes exactly two letters.

I strongly disagree with this assessment. This patch merely trades one set of issues for another one.

In particular, for pc machine types we'd end up producing bus=pci.0,addr=0x10 for slot 10 instead of bus=pci.0,addr=0xa, because the addr=0x part is baked into the template. So the QEMU logic David mentioned above wouldn't kick in at all.

More importantly, for q35 machines we'd start producing decimal bus numbers while still parsing the ones present in the original command line as hexadecimal, so things would stop lining up as soon as enough devices are present, meaning that the issue reported by Alona would still exist.

Oh, okay, sorry, I thought you and David agreed that it actually happens to work. But anyway, nice that it doesn't matter now. :)

Sorry, I missed that '0x' was in the template. Of course that can be very easily fixed.

...

...
...
As soon as you have something less qrappy we'll go with that (you don't even need to rebase, I'll revert this one on the tree first).

Patches fixing this issue, as well as a few additional ones, are now on the list. I'll follow up on that thread with some considerations related to testing the changes.

Thanks a lot!

I didn't push out commits or a release today because I hit a false positive with cppcheck 2.10 (cppcheck bisected but still trying to grasp the issue), so that will all be for the next week I guess.

-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson

830

Age (days ago)

834

Last active (days ago)

List overview

Download

11 comments

3 participants

participants (3)

Andrea Bolognani
David Gibson
Stefano Brivio

[PATCH] qrap: Pass PCI device numbers to qemu in base 10, not in base 16

tags

participants (3)