PCap-NG support in Wireshark and Tshark

Discussion:

Matthias Lang

2013-12-18 12:46:31 UTC

Hi,

Summary: I've stumbled on some unexpected details in how wireshark and
tshark handle PCap-NG. I'm not sure if they're bugs or
insufficient RTFM, so I thought I'd run them by here before
submitting bugs.

--------------------

I've checked out revision 54142 from subversion and built it on a
Debian AMD64 box. I.e. I think I'm looking at fresh code on a sane system.

1. The manpage (tshark.pod) for 'tshark' says reading from stdin isn't
allowed. But it actually works fine. Manpage says:

| =item -r E<lt>infileE<gt>
|
| Read packet data from I<infile>, can be any supported capture file format
| (including gzipped files). It's B<not> possible to use named pipes
| or stdin here!

cat /tmp/dumpcap.pcapng | ./tshark -r -

0 0.000000000 0 192.168.1.3 -> 255.255.255.255 UDP 211 ...
0 0.000027000 0 192.168.1.3 -> 255.255.255.255 UDP 235 ...

mkfifo /tmp/a_named_pipe
cat /tmp/dumpcap.pcapng > /tmp/a_named_pipe &
./tshark -r /tmp/a_named_pipe

0 0.000000000 0 192.168.1.3 -> 255.255.255.255 UDP 211 ...

I figure this can be fixed by just removing the word "not" from the
manpage.

Q: Ok if I submit a diff for my rewording?

2. tshark.pod says

| B<TShark>'s native capture file format is B<pcap> format

It might help someone if it said "B<Pcap-NG>". I'm not sure if
Pcap-NG is considered a separate format or a new version. It feels
like a separate format to me since it's totally incompatible.

Q: Ok if I submit a diff for my rewording?

3. Wireshark can read a Pcap from stdin, but not Pcap-NG.

./wireshark -r /tmp/dumpcap.pcapng
./wireshark -k -i - < /tmp/dumpcap.pcapng
cat /tmp/dumpcap.pcapng > /tmp/a_named_pipe &
wireshark -k -i /tmp/a_named_pipe
./tshark -r /tmp/dumpcap.pcapng -F pcap -w - | ./wireshark -k -i -

Q: is there a switch I'm missing or is this a bug?

4. The wireshark wiki says "if_tsresol" is supported, but only for
values 6 (microseconds) and 9 (nanoseconds).

Wireshark is better than that.

I verified that 0, 1, 2 and 3 (seconds, tenths of seconds,
hundreths, milliseconds) work in both tshark and
wireshark. milliseconds are important to me because the Pcap-NG
files I generate have millisecond timestamps.

I suspect all values work. I'm not good enough at subversion to
figure out since when. It might be commit r41455.

Wiki: http://wiki.wireshark.org/Development/PcapNg

Q: Ok if I edit the wiki to just say that it works? i.e. without
the limitation?

5. If an IDB in a Pcap-NG file contains both 'if_name' and
'if_description', wireshark only seems to display the
if_description; I can't figure out a way to see the name.

The way I see the 'if_name' is to open the file and go in to
'Statistics/Summary'.

Here's an example of each type of file:

if_name: http://corelatus.se/~matthias/if_name_only.pcapng
if_name and if_description: http://corelatus.se/~matthias/if_name_and_description.pcapng

And here's a screenshot of the "good" case, i.e. when you only
have the if_name field:

Loading Image...

I generated the Pcap-NG file, so it's possible I've stuffed up and
made a file that's somehow invalid. But it looks fine to me.

Q: Can I report this as a bug? (Or is it deliberate?)

6. I can't figure out a way to display the interface _name_ in
the Wireshark packet detail pane. Here's a screenshot where I've
got the interface _number_ displayed:

Loading Image...

I found an earlier discussion about this:

http://www.wireshark.org/lists/wireshark-users/201206/msg00082.html

Q: Is there a way to get at "frame.interface_name"? Both for display
and filtering. (last time, the answer was "not yet")

Q: Same question for tshark.

Background: I work with signalling on SS7 (telecom) networks. One
hardware capture device can capture signalling from hundreds of
interfaces.

Being able to view, filter and sort packets captured on multiple
interfaces is really useful, so I implemented support for writing
PCap-NG files instead of classic PCap. And now I'm exploring what
I can do with the result in Wireshark and tshark.

Matt
___________________________________________________________________________
Sent via: Wireshark-dev mailing list <wireshark-dev-IZ8446WsY0/***@public.gmane.org>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-request-IZ8446WsY0/***@public.gmane.org?subject=unsubscribe

Anders Broman

2013-12-29 10:15:19 UTC

Permalink

Post by Matthias Lang
Hi,
Summary: I've stumbled on some unexpected details in how wireshark and
tshark handle PCap-NG. I'm not sure if they're bugs or
insufficient RTFM, so I thought I'd run them by here before
submitting bugs.
--------------------
I've checked out revision 54142 from subversion and built it on a
Debian AMD64 box. I.e. I think I'm looking at fresh code on a sane system.
1. The manpage (tshark.pod) for 'tshark' says reading from stdin isn't
| =item -r E<lt>infileE<gt>
|
| Read packet data from I<infile>, can be any supported capture file format
| (including gzipped files). It's B<not> possible to use named pipes
| or stdin here!

cat /tmp/dumpcap.pcapng | ./tshark -r -

0 0.000000000 0 192.168.1.3 -> 255.255.255.255 UDP 211 ...
0 0.000027000 0 192.168.1.3 -> 255.255.255.255 UDP 235 ...

mkfifo /tmp/a_named_pipe
cat /tmp/dumpcap.pcapng > /tmp/a_named_pipe &
./tshark -r /tmp/a_named_pipe

0 0.000000000 0 192.168.1.3 -> 255.255.255.255 UDP 211 ...
I figure this can be fixed by just removing the word "not" from the
manpage.
Q: Ok if I submit a diff for my rewording?

Yes

Post by Matthias Lang
2. tshark.pod says
| B<TShark>'s native capture file format is B<pcap> format
It might help someone if it said "B<Pcap-NG>". I'm not sure if
Pcap-NG is considered a separate format or a new version. It feels
like a separate format to me since it's totally incompatible.
Q: Ok if I submit a diff for my rewording?

Yes

Post by Matthias Lang
3. Wireshark can read a Pcap from stdin, but not Pcap-NG.

Q: is there a switch I'm missing or is this a bug?

Bug or not yet implemented, open a bug.

Post by Matthias Lang
4. The wireshark wiki says "if_tsresol" is supported, but only for
values 6 (microseconds) and 9 (nanoseconds).
Wireshark is better than that.
I verified that 0, 1, 2 and 3 (seconds, tenths of seconds,
hundreths, milliseconds) work in both tshark and
wireshark. milliseconds are important to me because the Pcap-NG
files I generate have millisecond timestamps.
I suspect all values work. I'm not good enough at subversion to
figure out since when. It might be commit r41455.
Wiki: http://wiki.wireshark.org/Development/PcapNg
Q: Ok if I edit the wiki to just say that it works? i.e. without
the limitation?

Yes

Post by Matthias Lang
5. If an IDB in a Pcap-NG file contains both 'if_name' and
'if_description', wireshark only seems to display the
if_description; I can't figure out a way to see the name.
The way I see the 'if_name' is to open the file and go in to
'Statistics/Summary'.
if_name: http://corelatus.se/~matthias/if_name_only.pcapng
if_name and if_description: http://corelatus.se/~matthias/if_name_and_description.pcapng
And here's a screenshot of the "good" case, i.e. when you only
http://blog.corelatus.com/static/pcap_ng_wireshark_summary_screenshot.png
I generated the Pcap-NG file, so it's possible I've stuffed up and
made a file that's somehow invalid. But it looks fine to me.
Q: Can I report this as a bug? (Or is it deliberate?)

Bug or not yet implemented open a bug and include your test file.

Post by Matthias Lang
6. I can't figure out a way to display the interface _name_ in
the Wireshark packet detail pane. Here's a screenshot where I've
http://blog.corelatus.com/static/pcap_ng_wireshark_screenshot.png
http://www.wireshark.org/lists/wireshark-users/201206/msg00082.html
Q: Is there a way to get at "frame.interface_name"? Both for display
and filtering. (last time, the answer was "not yet")
Q: Same question for tshark.

Probebly not yet implemented open a bug so it won't be forgotten.

Post by Matthias Lang
Background: I work with signalling on SS7 (telecom) networks. One
hardware capture device can capture signalling from hundreds of
interfaces.
Being able to view, filter and sort packets captured on multiple
interfaces is really useful, so I implemented support for writing
PCap-NG files instead of classic PCap. And now I'm exploring what
I can do with the result in Wireshark and tshark.
Matt
___________________________________________________________________________
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev

___________________________________________________________________________
Sent via: Wireshark-dev mailing list <wireshark-dev-IZ8446WsY0/***@public.gmane.org>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-request-IZ8446WsY0/***@public.gmane.org?subject=unsubscribe

Guy Harris

2013-12-29 11:41:05 UTC

Permalink

Post by Matthias Lang
1. The manpage (tshark.pod) for 'tshark' says reading from stdin isn't
| =item -r E<lt>infileE<gt>
|
| Read packet data from I<infile>, can be any supported capture file format
| (including gzipped files). It's B<not> possible to use named pipes
| or stdin here!

That text might have been historically correct; some changes have been made to libwiretap to attempt to make it work, at least with some capture file formats:

r43102 | guy | 2012-06-05 00:24:17 -0700 (Tue, 05 Jun 2012) | 9 lines

From Jakub Zawadzki: when seeking backwards, if the seek will put you at
a position that's in our data buffer, just reposition within the
buffer, don't do any seeks or I/O on the underlying file. This lets us
do some backwards seeking on a pipe, to allow the rewind-and-try scheme
we use to try to identify capture file types to work, at least for some
capture file formats (those that have magic numbers at the beginning or
have heuristics that don't require much data), on pipes, allowing, for
example, TShark to read those formats from a pipe.

but, as the comment suggests, if more data than fits in the data buffer is read by a file open routine that's called before the routine that recognizes the file's type, this will *not* work if the file is being read from a pipe.

Fortunately, both pcap and pcap-ng formats have magic numbers near the beginning, and their open routines are called before other ones (as they're the native formats for Wireshark), so reading pcap or pcap-ng files from a pipe will probably work (although the pcap file reader does some additional reading to try to handle some non-standard pcap formats, and if *that* reads more than will fit in a buffer, the pcap-ng reader won't get to read the file as the seek-to-the-beginning will fail on a pipe).

So it's more like "it might, or might not, be possible to read from a pipe here, depending on the file type and the contents of the file".

It's a separate format. TShark doesn't have the equivalent of dumpcap's -P flag to tell it to use pcap rather than pcap-ng, so (at least until such an option is added) it's probably best to say its native format is pcap-ng.
___________________________________________________________________________
Sent via: Wireshark-dev mailing list <wireshark-dev-IZ8446WsY0/***@public.gmane.org>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-request-IZ8446WsY0/***@public.gmane.org?subject=unsubscribe

Jakub Zawadzki

2013-12-29 14:11:40 UTC

Permalink

Post by Guy Harris

[...]
Fortunately, both pcap and pcap-ng formats have magic numbers near the beginning, and their open routines are called before other ones (as they're the native formats for Wireshark), so reading pcap or pcap-ng files from a pipe will probably work (although the pcap file reader does some additional reading to try to handle some non-standard pcap formats, and if *that* reads more than will fit in a buffer, the pcap-ng reader won't get to read the file as the seek-to-the-beginning will fail on a pipe).
So it's more like "it might, or might not, be possible to read from a pipe here, depending on the file type and the contents of the file".

It doesn't always work with pcap-ng, for example check bug #9533 [1].

[1] https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=9533

Kuba.
___________________________________________________________________________
Sent via: Wireshark-dev mailing list <wireshark-dev-IZ8446WsY0/***@public.gmane.org>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-request-IZ8446WsY0/***@public.gmane.org?subject=unsubscribe

Guy Harris

2013-12-29 19:21:04 UTC

Permalink

Post by Guy Harris
So it's more like "it might, or might not, be possible to read from a pipe here, depending on the file type and the contents of the file".

Note also that there are file formats that *cannot* be read from a pipe, as even reading the file once, from the first packet to the last, requires seeking forward and then backward. This includes "NetXRay" format (as used by the old network analyzer of the same name, as well as by the Windows Sniffer applications) and Network Monitor format.

___________________________________________________________________________
Sent via: Wireshark-dev mailing list <wireshark-dev-IZ8446WsY0/***@public.gmane.org>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-request-IZ8446WsY0/***@public.gmane.org?subject=unsubscribe

Guy Harris

2013-12-29 22:45:33 UTC

Permalink

Post by Guy Harris
So it's more like "it might, or might not, be possible to read from a pipe here, depending on the file type and the contents of the file".

...and it is *not* possible to read from a pipe if you hand tshark the -2 flag, causing it to make two passes through the file.

___________________________________________________________________________
Sent via: Wireshark-dev mailing list <wireshark-dev-IZ8446WsY0/***@public.gmane.org>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-request-IZ8446WsY0/***@public.gmane.org?subject=unsubscribe

Guy Harris

2013-12-29 22:41:46 UTC

Permalink

Post by Matthias Lang
3. Wireshark can read a Pcap from stdin, but not Pcap-NG.

./wireshark -r /tmp/dumpcap.pcapng
./wireshark -k -i - < /tmp/dumpcap.pcapng
cat /tmp/dumpcap.pcapng > /tmp/a_named_pipe &
wireshark -k -i /tmp/a_named_pipe

That's better stated as "Wireshark can capture from a pipe if handed a pcap file, but not if handed a pcap-ng file"; Wireshark can't *read* from a pipe, but it can *capture* from a pipe and write the packets to a file (the packets *have* to be in a file in Wireshark, so that it can seek and re-read arbitrary packets).

And there's currently no code in dumpcap (which is what's reading from the pipe) to handle pcap-ng files, so, no, neither Wireshark nor TShark nor dumpcap can capture from a pipe if what's being written to the pipe is a pcap-ng file.

___________________________________________________________________________
Sent via: Wireshark-dev mailing list <wireshark-dev-IZ8446WsY0/***@public.gmane.org>
Archives: http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
mailto:wireshark-dev-request-IZ8446WsY0/***@public.gmane.org?subject=unsubscribe