INTERIM_MEETING_REPORT_ Reported by Lou Berger/Bolt Beranek and Newman Minutes of the Internet Stream Protocol V2 Working Group (ST2) The ST-II Working Group held two days of meetings on 8 and 9 February. The meetings were held at George Mason University (GMU) and were hosted by Professor Mark Pullen. Agenda The goal of the meeting was to cover the following items: o State diagrams o ST-II service o Elimination of HIDs o Elimination of options/parameters o Groups of streams o Receiver-oriented connections o RFC structure The following items were also identified as needing attention but the group did not feel that there would be enough time to cover them: o Routing o FlowSpec o ST-II MIB o ST-II over various networks (ATM, Ethernet, PPP, FDDI, etc.) o Alternate data packet headers o Sub-flows/heterogeneous flows o Reason codes State Diagram Review Murali Rajagopal presented a series of state diagrams based on his reading of RFC 1190. (His slides will be added to the ST-II Working Group archive.) He presented a model that contained five state machines: 1. FSM at origin 2. FSM for the origin-side of a router 3. FSM for the target-side of a router 4. FSM to coordinate/synchronize the router origin-side and target-side FSMs 5. FSM at target He also listed what he saw as unresolved issues in the RFC: o When acknowledgements on connect and other messages should be sent o When can data be sent (after connect issued or after accept received) o HID negation is complex and may be flawed o The state transition out of the ``connecting'' state is unresolved when a connect message goes unanswered. o Is the complexity of full duplex needed o The RFC should include state tables o Restriction on possible state transitions o It is possible to receive disconnects in the middle of stream establishment o It is possible to receive multiple connects at any point in time o There are mistakes in the current RFC (uses wrong message type) Some time was spent reviewing and discussing the state diagrams. The group felt that there may be a problem in the presented synchronization FSM with handling refuses. It was agreed that this was probably a problem in the RFC that was propagated to the FSM diagram. Murali presented tables listing which messages may be issued as requests and which as responses. Murali ended with a list of specific issues with the current RFC: o HID-CHANGE and HID-CHANGE-REQUEST messages may not converge on an HID o DISCONNECT message appears to apply to intermediate agents o Contradiction to the use of DISCONNECT message (page 114) downstream machine sends a DISCONNECT instead of REFUSE, contradicts page 79, page 110 o HID-REJECT and HID-APPROVE are valid responses to change? o What is REROUTE? (REROUTE has now been removed) o There are ``criss-cross'' states o REFUSE used inconsistently ST-II Service Luca lead a discussion on the user's view of the service provided by ST-II. This included a discussion on when a user can send data (i.e. after sending connect, or after first accept) and states seen by the user. The states were documented by Claudio Topolcic. A state diagram will be included in the Internet-Draft. This was followed by a proposal to limit the combination of messages that can be sent at one time, and thereby to limit state transitions. The following table represents the agreed upon allowed actions per state. _______________________________________________________________ |_State______________|Allowed_Transaction______________________| | | | | IDLE |CONNECT --> CONNECTION PENDING | | |(initiated by API at origin) | | | | | CONNECTION PENDING |TIMEOUT --> DROP --> ESTABLISHED | | |or CLOSE --> IDLE | | |(IDLE when all refused) | | |(ESTABLISHED when at least 1 and accept) | | |ALL REFUSE --> IDLE | | |ALL ACCEPT --> ESTABLISHED | | |LAST RESP + at least 1 ACCEPT | | |--> ESTABLISHED | | |ACCEPT | REFUSE --> CONNECTION PENDING | | |DATA can be sent at any time | | | | | ESTABLISHED |{ADD, DROP, CHANGE, DATA, CLOSE} | | |DROP --> ESTABLISH | | |CLOSE --> IDLE | | |API ISSUES CHANGE --> CHANGE | | |CHANGE-REQUEST translated by | | |API to CHANGE or ignored | | |REFUSED --> ESTABLISHED | | |ALL REFUSED --> IDLE | | CHANGE |DATA | | |REFUSES may be received | | | | | ADD TARGET |DATA | |____________________|REFUSES_may_be_received__________________| The discussion on ST-II service closed with a review of open issues: o Steve Jackowski volunteered to write a description of where ST-II fits in from the application perspective. o Should HID negation be eliminated (replaced)? o Handling of heterogeneous receivers and interpretation of heterogeneous FlowSpecs. o Handling of additions of more targets than can be contained in an MTU connect message. Options: - SCMP fragmentation - issue multiple connects There was a proposal to implement a rudimentary fragmentation capability for ST control packets only (not data). Even a simple send/acknowledgement scheme could be considered. o Full duplex complicates the state machine and when data can be sent. It was voted at the last meeting to drop FDX, so it is dropped! o Handling of stream preemption. o Can a router issue a CHANGE-REQUEST? Receiver Initiated Connections The issue of changing ST-II service to support receiver-initiated connections was discussed next. The general consensus was that it should be added, but only if it is easy. In order to evaluate complexity, the group discussed some details. A `join' message was defined as having three components: 1. ST connection name 2. Origin's IP address 3. Desired QOS The way and order by which a receiver can initiate a connection was defined as follows: 1. Origin creates connection to 0 or more targets. In doing so it is assigned an ST name. 2. Target gets ST name out of band (maybe one RTT). 3. Target sends JOIN request, request propagates to origin. 4. First agent that is encountered that contains matching connection sends CONNECT. If the connection previously had no targets, the origin agent is the one that sends the connect. Steps 3 and 4 are repeated for every receiver-initiated connection. Several outstanding points were then discussed. 1) The first item was rerouting after a failure. The issue is that in order for an ST agent to attempt stream recovery it must have a full list of all targets. This implies that if automatic stream recovery is desired, all upstream agents must be notified of a successful join. 2) This leads to the second point, that maintaining full target lists limits the maximum size of an ST-II connection. This is due to message overhead and required state table space requirements. This implies that upstream agents should not be notified of a successful join. 3) The last point was that without origin notification of successful joins, there is no limiting who may join a connection. This lack of access control may be unacceptable to some applications. The group reviewed IBM's proposed solution to these issues. In its paper on receiver-initiated connections, IBM proposed solving the listed issues by defining per-connection classes of services. (The paper is in ibminet.awdpa.ibm.com:pub/st/t2-recv.ps.) Each class defines how receiver-initiated connections are handled. Four classes were defined as follows: 1. Sender initiated: Nobody is allowed to join, all joins are refused. 2. Ask application: Poll application prior to allowing each join. The group felt that this should be done out of band and should not be included. 3. Any with notification: Allow any agent to join a connection. Upstream agents are notified upon successful join. This permits each upstream agent to maintain a full target list. 4. Any, no notification: Allow any agent to join a connection. Upstream agents are not notified of any joins. Full target lists are not maintained. The group then discussed if ST-II name registration took place via the creation of a connection with no targets or via a special name registration process. The issue is when is the ``established'' state entered: is it after name registration or after the first join request? The group agreed that this is a special case only effecting the origin. General consensus was that there should be no special name registration function, and to support this via connects with zero targets. After such a connect, the state will be ``established.'' The issue of all agents maintaining full target lists was again raised. It was pointed out that without full target lists, specific target disconnects would not work. One of the GMU participants suggested that this problem can be solved via flooding (forward the connect) to all downstream next hop agents. This lead to more discussion on the value of receiver initiated connections. The group decided to chart out the implications of IBM's different classes of service. The two axes are ``access control'' and ``maintained state.'' Explanations are listed below that table. _______________________________________________ | NOBODY Ask Join Join | |_State___________Origin__w/notify__w/o_notify_| | Next Hop | |_Only_______Y1_____Y2_______Y3_________Y7_____| | full | |_Target_____Y4_____Y5_______Y6_________NA_____| Y1 Implies that individual drops cannot be handled (maybe by flooding). Only refuses and closes can be handled. No recovery, no individual QOS changes. The value of this is that it limits state space in intermediate hops. Is this worth anything? Y2 Implies that individual drops cannot be handled (maybe by flooding). Only refuses and closes can be handled. No recovery, no individual QOS changes. The value of this is that it limits state space in intermediate hops. Is this worth anything? Probably not (same as Y2). Y3 Implies that individual drops cannot be handled (maybe by flooding). Only refuses and closes can be handled. No recovery, no individual QOS changes. The value of this is that it limits state space in intermediate hops and shortens connect time. Is this worth anything? Probably not. Y4 Full target list is maintained at all agents so all requests and recovery can be supported. Matches current ST-II definition. Y5 Full target list is maintained at all agents so all requests and recovery can be supported. Is similar to current ST-II definition, but receivers initiate connection and origin can control access prior to successful join. Y6 Full target list is maintained at all agents so all requests and recovery can be supported. Is similar to current ST-II definition, but receivers initiate connection. Y7 Implies that individual drops cannot be handled (maybe by flooding). Only refuses and closes can be handled. No recovery, no individual QOS changes. The value of this is that it limits state space in intermediate hops. Claudio Topolcic suggested that Y1 through Y3 have little value since agents are involved in the connection but do not take any real actions. The group agreed that if state was not going to be maintained, only case Y7 makes sense. Claudio suggested that Y4 through Y6 were the same. It was agreed that Y6 differed in that connection time was shortened, and that this may have value in some cases. This left Y4 and Y5. Some group members felt that they were separate cases and should be included. Other members felt that Y5 was not different than Y4 used in conjunction with an out of band message sent requesting a connect by the receiver. The group agreed with the latter. Given the discussion, the above table was simplified into the following: ______________________________________ | Maintained |Attributes | | State |Access-control | |_____________|security_______________| | Next hop |Join w/o notify | |_only________|_______________________| | Full Target |no Join w/ | |_List________|join___________notify__| The following message types were decided to be needed: o JOIN-REQ o JOIN-REJECT previous hop will issue CONNECTs and target will answer with ACCEPTs. o JOIN-NOTIFY looks like accepts, acts like an accept. The issue was raised if a JOIN-NOTIFY message should be used or if an `unsolicited' ACCEPT should be used. It was agreed to leave this detail for the future. The group then discussed mapping the processing of the new messages into existing states: o Notify may be processed in all states o JOIN-REQ may only be processed when ESTABLISHED. (They are queued when not in established and handled after transition into established or idle.) o When ESTABLISHED and ALL REFUSED when in ``join possible'' state go to ESTABLISHED, null connection. Mark Pullen then asked if the group should be spending time working on receiver-initiated. The group felt that this was a minor modification (given IBM's running implementation experience), and Luca felt this fit in to the charter. The group also thought the issue may be revisited based on prototype implementation experience. The relationship to RSVP was also brought up. The group discussed this for a bit and concluded that ST experience should not be lost. That ST features should be moved into RSVP for the future, and that this working group should be focused on interoperability, product delivery, and ease of implementation. This migrated into a general discussion on what revised ST-II should be and the definition of interoperability. The group agreed that interoperable means interoperability between different implementations of new versions and not with current (RFC 1190) versions. Elimination of HIDs/VLIDs What should be done with VLIDs? Steve Jackowski asked if we can be IP address independent? Lots of discussion, should ST-II be protocol-family independent. This discussion was tabled for the future. Proposal: replace with 48-bit SIDs, 16-bit unique ID + IP address. How does this affect performance? Most thought not a great deal. Conclusion: replace ST stream name with SID. Can all VLIDs be replaced with new 48-bit SID? Looks like VLIDs can be replaced by the combination of the existing ST name (SID + time-stamp) and existing originator (sender) field (page 77). Conclusion: replace HIDs, ST name, and VLIDs with SID. Is it possible for SCMP to setup resources for arbitrary data formats? Digression on heterogeneous flows. Do we want to include sender information in data packets for debugging? o Now will know origin but may not know sender, is this enough? - Can talk to origin but not sender - Can generate an error but maybe not a refuse o General feeling is origin is enough, especially considering: - Keep alive mechanism - Maybe add a new error message message to be sent to origin Review of Inclusions and Exclusions Eliminated: o Time-stamp o Target list parameter (use target lists) o Reverse charge o Full duplex** (Luca may want to reopen based on group meeting) o Point to point o Errored PDU - page 70 o HID negotiation Modified: o replaced HIDs, VLIDs, ST name with SID Added: o Join Issues: o Error reason codes - maybe add fatal bit o Notify, status, hello messages o FlowSpec o IP Encapsulation o Preemption o Groups (List from above) o Routing o Heterogeneous flows Groups of Streams Used to associate attributes across multiple connections. Typical example, sharing bandwidth (e.g. multi-way video conferencing). Relationship: o resources (bandwidth, addresses) o interdependent streams (preemption/fate sharing) o routing (fate sharing/(in)dependence) o open/closed groups G = Streams + relationship + attributes (e.g. group access control). May need new options parameter giving relationship + attributes. When does a connection get added to a group? It is ``easier'' if it happens at stream creation and persists for life of connection. Should there be an enforced limit? There was a lot of discussion on what is bandwidth sharing. Proposal: allocate max stream size, or multiplier of it. There was discussion on what is our objective. Define group mechanism. Define relationships that can be identified (and are thought to be usable). Action: need to flush out relationships. Proposed resource allocation sharing parameter: allocation is based on per connection ``reserved bandwidth'' multiplied by the parameter. Open/Closed groups: decided to only have open. Closed is ugly and requires new mechanisms. Why would ``routing dependency'' be used: fate sharing, similar FlowSpec do we need this relation ship? Does not look like it. Want shared resources (bandwidth, addresses). Steve wants interdependent streams: (fate sharing) for full duplex connections and preemption. Is there a limit on how many groups may a stream be associated with? Looks like multiple would be useful (shared bandwidth+fate sharing). What exactly is being shared, do we share resources that are control via a single agent or across the whole subnet? Lou wants across the whole sub-net. Everyone thinks this is very hard, Luca says that IBM does this by using a central manager (ala ATM ARP server). Lou considers this important for the wide area. Want parameter indicates to the sub-net layer to optimize use of multicast addresses, and/or share bandwidth between different ``sources?'' Claudio suggests that if its not going to be implemented do not include it in the specification. Do we want to drop groups? Open discussions: Discussion on sending data once first ACCEPT received, problem of what to do when subsequent ACCEPTS come in with smaller FlowSpec. Intermediate (or any) agents will not forward data until 1) resources are allocated or 2) ACCEPT received. Undocumented implementation issue: all implementors do not propagate CONNECT until local resources allocated. Discussion on handling of higher offer load than FlowSpec. General discussion on requirement for definition of how resource enforcement is done. Alternatives: when exceeds ! random traffic dropped (current implementations) Claudio says that this unspecified behavior is totally unacceptable. o Proposal 1: FlowSpec will be met, excess traffic may be dropped. If dropped, lower priority packets are dropped first. o Proposal 2: Allow mechanism that permits a customized dropping algorithm. o Group decided that this needs resolution. Agree that will use ST priority bits. o Proposal 3: Dropping algorithm based on FlowSpec. Default behavior with default FlowSpec is FlowSpec will be met, excess traffic may be dropped or delivered outside FlowSpec. If dropped (or delivered late), lower priority packets are dropped (or delivered late) first. 0=drop first, 7 = drop last. The conclusion was to do Proposal 3. There was discussion of definition of priority. Issue: Do higher priority packets get processed first? Luca thinks this is evil and should be avoided and that processing order should not be affected and only dropping should be affected. Question: Does this preclude misordering. Claudio says no, but want nondeterministic misordering ok. Need to think about regulation policy. RFC Structure 1. Introduction o Position technology (why ST-II exists) o Components in reservation world o Relationships o ST-II's role (what it does) o Co-requisite components (other modules required for ST-II to operate) o ST-II and applications o ST-II and other protocols 2. Basic Concepts (High level definition) o Terminology o Streams - types, Connection process, FlowSpec o Multicast o Differences form other connection oriented protocols o Out of band issues 3. High level functional description with examples o Includes: high level services 4. SCMP functional description 5. State machines and tables 6. ST-II PDU formats o Note no acknowledgements 7. FlowSpec o Routing o IP encapsulation 8. Index Separate RFC ST-II over sub-nets specifics examples of nets (ether, FDDI, ATM, PPP, etc.). An issue is how to get ``small'' paper. Maybe put 1-3 in a separate document which gives high level concepts. The proposal is to make 1-3 a separate document from 4-6. This was agreed. It is possibe that 6 should be a separate document. There was a lot of discussion on who does what. Components: o IP o Routing o Resource managers - Admission control - Allocation - Scheduling o Subnet interface (includes subnet resource manager) o Resource enforcement Heterogeneous Flows Can be done via multiple streams or by heterogeneous flows. Useful for hierarchical coding where lower priority packets are discarded at network ``choke'' points. Luca proposes heterogeneous connection with FlowSpec that describes requirements of sub flows. Data packets has sub-flow identifier + drop priority bits. A disadvantage is complicated FlowSpec. An advantage is graceful handling of hierarchical flows. Claudio proposes the use of a single FlowSpec and the use of a drop priority field. A disadvantage is that it may waste bandwidth. An advantage is that there are no extra mechanisms. Should this be an option? Should there be any options? No options are wanted. Maybe there should be an options capability but with none defined. Also need to consider implications of case where subsequent targets (connects) are able to receive larger sub-set of flows. o Reason Codes - each person should write up their favorites so all can compare them. o Tunneling - someone should write up a proposal. o Different packet headers - this needs more study. First, someone should ensure that we are not stepping on RSVP people's toes. o Routing - we need to provide guidance to the routing module, such as what ST-II expects from the routing module. o FlowSpec - pass this one for today. Everybody should send in their FlowSpec, plus the ones that are in the literature. o MIB - in progress. Luca and Chip will publish theirs. o State machines - Murali will write up modified state machines for the simplified protocol that we came up with. o ST2 over Foo subnet - Luca wants help because different people are using different subnets. Each proponent should write up ``ST2 over their favorite net.'' Will probably need some coordination up front. Claudio and Steve will each write up their proposals to support heterogeneous receivers. Attendees Lou Berger lberger@bbn.com Luca Delgrossi luca@ibmpa.awdpa.ibm.com Abe Drier adrier@mason1.gmu.edu Jeffrey Dunn dunn@neptune.nrl.navy.mil Eric Herrman eherrman@mason1.gmu.edu Steven Jackowski stevej@syzygycomm.com Cynthia Martin martin@neptune.nrl.navy.mil Mark Pullen mpullen@cs.gmu.edu Murali Rajagopal murali@mitre.org Claudio Topolcic topolcic@bbn.com Ann Tso tsoa@cc.ims.disa.mil